[
https://issues.apache.org/jira/browse/IMPALA-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18007525#comment-18007525
]
Quanlong Huang commented on IMPALA-14227:
-----------------------------------------
Patch is under review: https://gerrit.cloudera.org/c/23174
> In HA failover, passive catalogd should apply pending HMS events before being
> active
> ------------------------------------------------------------------------------------
>
> Key: IMPALA-14227
> URL: https://issues.apache.org/jira/browse/IMPALA-14227
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Blocker
>
> After IMPALA-14074, the passive catalogd can have a warmed up metadata cache
> during failover (with catalogd_ha_reset_metadata_on_failover=false). However,
> it could still have pending HMS events that are not applied and so is using a
> stale metadata cache.
> For instance, the active catalogd creates a table and then crash. The passive
> catalogd should apply the CREATE_TABLE events before being active. Otherwise,
> Impala queries might see stale metadata in a while (until the new catalogd
> catch up with HMS events generated by the previous active catalogd).
> There is a test failure caused by this:
> {code:python}
> custom_cluster/test_catalogd_ha.py:540: in
> test_warmed_up_metadata_after_failover
> latest_catalogd = self._test_metadata_after_failover(unique_database,
> True)
> custom_cluster/test_catalogd_ha.py:584: in _test_metadata_after_failover
> self.execute_query_expect_success(self.client, "describe %s.tbl" %
> unique_database)
> common/impala_test_suite.py:1121: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:1131: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:1294: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:687: in execute
> cursor.execute(sql_stmt, configuration=self.__query_options)
> ../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:392:
> in execute
> configuration=configuration)
> ../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:443:
> in execute_async
> self._execute_async(op)
> ../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:462:
> in _execute_async
> operation_fn()
> ../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:440:
> in op
> run_async=True)
> ../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:1324:
> in execute
> return self._operation('ExecuteStatement', req, False)
> ../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:1244:
> in _operation
> resp = self._rpc(kind, request, safe_to_retry)
> ../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:1181:
> in _rpc
> err_if_rpc_not_ok(response)
> ../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:867:
> in err_if_rpc_not_ok
> raise HiveServer2Error(resp.status.errorMessage)
> E HiveServer2Error: Query eb405217bbb418ee:a1033c0000000000 failed:
> E AnalysisException: Could not resolve path:
> 'test_warmed_up_metadata_after_failover_452d93b4.tbl'{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
