Quanlong Huang created IMPALA-14227:
---------------------------------------
Summary: In HA failover, passive catalogd should apply pending HMS
events before being active
Key: IMPALA-14227
URL: https://issues.apache.org/jira/browse/IMPALA-14227
Project: IMPALA
Issue Type: Bug
Reporter: Quanlong Huang
Assignee: Quanlong Huang
After IMPALA-14074, the passive catalogd can have a warmed up metadata cache
during failover (with catalogd_ha_reset_metadata_on_failover=false). However,
it could still have pending HMS events that are not applied and so is using a
stale metadata cache.
For instance, the active catalogd creates a table and then crash. The passive
catalogd should apply the CREATE_TABLE events before being active. Otherwise,
Impala queries might see stale metadata in a while (until the new catalogd
catch up with HMS events generated by the previous active catalogd).
There is a test failure caused by this:
{code:python}
custom_cluster/test_catalogd_ha.py:540: in
test_warmed_up_metadata_after_failover
latest_catalogd = self._test_metadata_after_failover(unique_database, True)
custom_cluster/test_catalogd_ha.py:584: in _test_metadata_after_failover
self.execute_query_expect_success(self.client, "describe %s.tbl" %
unique_database)
common/impala_test_suite.py:1121: in wrapper
return function(*args, **kwargs)
common/impala_test_suite.py:1131: in execute_query_expect_success
result = cls.__execute_query(impalad_client, query, query_options, user)
common/impala_test_suite.py:1294: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:687: in execute
cursor.execute(sql_stmt, configuration=self.__query_options)
../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:392:
in execute
configuration=configuration)
../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:443:
in execute_async
self._execute_async(op)
../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:462:
in _execute_async
operation_fn()
../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:440:
in op
run_async=True)
../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:1324:
in execute
return self._operation('ExecuteStatement', req, False)
../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:1244:
in _operation
resp = self._rpc(kind, request, safe_to_retry)
../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:1181:
in _rpc
err_if_rpc_not_ok(response)
../infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:867:
in err_if_rpc_not_ok
raise HiveServer2Error(resp.status.errorMessage)
E HiveServer2Error: Query eb405217bbb418ee:a1033c0000000000 failed:
E AnalysisException: Could not resolve path:
'test_warmed_up_metadata_after_failover_452d93b4.tbl'{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
