[
https://issues.apache.org/jira/browse/IMPALA-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18042069#comment-18042069
]
Quanlong Huang commented on IMPALA-14582:
-----------------------------------------
The coordinator logs of that statement:
{noformat}
I20251201 19:28:07.531167 72684 Frontend.java:2445]
ea4e44b926dd9971:e0063c5a00000000] Analyzing query: show create table
test_show_create_table_with_stats_5f4ae381.t_part_stats WITH STATS db: default
...
W20251201 19:28:07.548882 72684 CatalogdMetaProvider.java:531]
ea4e44b926dd9971:e0063c5a00000000] Catalog object TCatalogObject(type:TABLE,
catalog_version:50379,
table:TTable(db_name:test_show_create_table_with_stats_5f4ae381,
tbl_name:t_part_stats)) changed version from 50379 to 50381 while fetching
metadata
I20251201 19:28:07.549014 72684 jni-util.cc:321]
ea4e44b926dd9971:e0063c5a00000000]
org.apache.impala.catalog.local.InconsistentMetadataFetchException: Catalog
object TCatalogObject(type:TABLE, catalog_version:50379,
table:TTable(db_name:test_show_create_table_with_stats_5f4ae381,
tbl_name:t_part_stats)) changed version between accesses.
at
org.apache.impala.catalog.local.CatalogdMetaProvider.sendRequest(CatalogdMetaProvider.java:535)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadPartitionsFromCatalogd(CatalogdMetaProvider.java:1121)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadPartitionsByRefs(CatalogdMetaProvider.java:1039)
at
org.apache.impala.catalog.local.LocalFsTable.loadPartitions(LocalFsTable.java:473)
at
org.apache.impala.analysis.ToSqlUtils.appendPartitionStatements(ToSqlUtils.java:770)
at
org.apache.impala.analysis.ToSqlUtils.getCreateTableWithStatsSql(ToSqlUtils.java:678)
at
org.apache.impala.service.JniFrontend.showCreateTable(JniFrontend.java:613){noformat}
Coordinator should catch InconsistentMetadataFetchException and retry.
On the other hand, I'm curious why the table changed in catalogd. Checked the
catalogd logs to see how the table version changes from 50379 to 50381, I found
it's due to HMS event processing:
{noformat}
I20251201 19:28:07.541288 150 CatalogServiceCatalog.java:1382] Could not find
version 50361 in in-flight event list of table
test_show_create_table_with_stats_5f4ae381.t_part_stats
I20251201 19:28:07.541352 150 MetastoreEvents.java:936] EventId: 40529
EventType: ALTER_TABLE Target:
test_show_create_table_with_stats_5f4ae381.t_part_stats. Skipping reloading of
file metadata for table test_show_create_table_with_stats_5f4ae381.t_part_stats
since SD and whitelistedTblProperties has not changed.
I20251201 19:28:07.541396 150 CatalogServiceCatalog.java:3195] Refreshing
table metadata: test_show_create_table_with_stats_5f4ae381.t_part_stats
I20251201 19:28:07.544323 150 HdfsTable.java:1256] Reloading metadata for
table definition and all partition(s) of
test_show_create_table_with_stats_5f4ae381.t_part_stats (ALTER_TABLE event
40529)
I20251201 19:28:07.544654 150 HdfsTable.java:1923] Loaded 3 columns from HMS.
Actual columns: 3
I20251201 19:28:07.548062 150 HdfsTable.java:3003] Load Valid Write Id List
Done. Time taken: 1.322us
I20251201 19:28:07.548465 150 HdfsTable.java:1305] Incrementally loaded table
metadata for: test_show_create_table_with_stats_5f4ae381.t_part_stats
I20251201 19:28:07.548533 150 HdfsTable.java:3095] Setting the catalog
version of HdfsTable@c0afb98d
test_show_create_table_with_stats_5f4ae381.t_part_stats to 50381{noformat}
Operations on this table are all done in Impala. I think it's a bug that
ALTER_TABLE event 40529 is not treated as a self-event.
[~hemanth619], [~VenuReddy] Do you have time to take a look? We might need to
fire a separate JIRA for the self-event detection bug.
> test_show_create_table_with_stats is flaky
> ------------------------------------------
>
> Key: IMPALA-14582
> URL: https://issues.apache.org/jira/browse/IMPALA-14582
> Project: IMPALA
> Issue Type: Bug
> Components: Test
> Affects Versions: Impala 5.0.0
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Major
> Labels: broken-build
> Attachments:
> catalogd.6f230ec72740.impala.log.INFO.20251201-183755.1.gz
>
>
> test_show_create_table_with_stats is flaky can be flaky due to inconsistent
> metadata between after table queries in test file.
> It recently failed in
> [https://jenkins.impala.io/job/ubuntu-20.04-dockerised-tests/4296/testReport/junit/metadata.test_show_create_table/TestShowCreateTable/test_show_create_table_with_stats__table_format__text_none_____protocol__hs2__/]
>
> {code:java}
> Stacktrace
> metadata/test_show_create_table.py:106: in test_show_create_table_with_stats
> self.__run_show_create_table_with_stats_test_case(
> metadata/test_show_create_table.py:249: in
> __run_show_create_table_with_stats_test_case
> result = self.__exec(test_case.show_create_table_sql + " WITH STATS")
> metadata/test_show_create_table.py:263: in __exec
> return self.execute_query_expect_success(self.client, sql_str)
> common/impala_test_suite.py:1126: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:1136: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:1299: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:687: in execute
> cursor.execute(sql_stmt, configuration=self.__query_options)
> ../infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/impala/hiveserver2.py:391:
> in execute
> self.execute_async(operation, parameters=parameters,
> ../infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/impala/hiveserver2.py:443:
> in execute_async
> self._execute_async(op)
> ../infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/impala/hiveserver2.py:462:
> in _execute_async
> operation_fn()
> ../infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/impala/hiveserver2.py:438:
> in op
> op = self.session.execute(self._last_operation_string,
> ../infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/impala/hiveserver2.py:1324:
> in execute
> return self._operation('ExecuteStatement', req, False)
> ../infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/impala/hiveserver2.py:1244:
> in _operation
> resp = self._rpc(kind, request, safe_to_retry)
> ../infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/impala/hiveserver2.py:1181:
> in _rpc
> err_if_rpc_not_ok(response)
> ../infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/impala/hiveserver2.py:867:
> in err_if_rpc_not_ok
> raise HiveServer2Error(resp.status.errorMessage)
> E impala.error.HiveServer2Error: Query ea4e44b926dd9971:e0063c5a00000000
> failed:
> E InconsistentMetadataFetchException: Catalog object
> TCatalogObject(type:TABLE, catalog_version:50379,
> table:TTable(db_name:test_show_create_table_with_stats_5f4ae381,
> tbl_name:t_part_stats)) changed version between accesses.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
