[
https://issues.apache.org/jira/browse/IMPALA-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609564#comment-17609564
]
ASF subversion and git services commented on IMPALA-11508:
----------------------------------------------------------
Commit aaf6fdc645b2f67e67f009c8e6dcf3ce25b22c98 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=aaf6fdc64 ]
IMPALA-11508: Deflake test_expire_snapshots
Before this patch test_expire_snapshots failed frequently. The patch is
only a few lines of code, but there are some subtleties here:
IcebergCatalogOpExecutor.alterTableExecute() didn't use Iceberg
transactions to carry out the operation. This means that
expireSnapshots() resulted in an ALTER TABLE operation on its own
which we also got during event processing.
Because this ALTER TABLE event didn't had the catalog version set we
didn't recognized it as a self-event. This caused unnecessary table
reloads during the tests which manifested in
InconsistentMetadataFetchException: "... table ... changed version
between accesses" errors.
With this patch IcebergCatalogOpExecutor.alterTableExecute() takes
an Iceberg transaction object and invokes expireSnapshots() in the
context of this Iceberg transaction. This Iceberg transaction also
sets table properties "impala.events.catalogServiceId" and
"impala.events.catalogVersion". And because everything happens in
a single Iceberg transaction we only create a single ALTER TABLE
which we can recognize during event processing (based on the
table properties), avoiding unnecessary table reloads.
Testing:
* executed test_expire_snapshots in a loop
Change-Id: I6d82c8b52466a24af096fe5fe4dbd034a1ee6a15
Reviewed-on: http://gerrit.cloudera.org:8080/19036
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Iceberg test test_expire_snapshots is flaky
> -------------------------------------------
>
> Key: IMPALA-11508
> URL: https://issues.apache.org/jira/browse/IMPALA-11508
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: broken-build, impala-iceberg
>
> h2. Stacktrace
> {noformat}
> query_test/test_iceberg.py:104: in test_expire_snapshots
> impalad_client.execute(insert_q)
> common/impala_connection.py:212: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:189: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:359: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:522: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> E MESSAGE: AnalysisException:
> org.apache.impala.catalog.TableLoadingException: Error opening Iceberg table
> 'test_expire_snapshots_e488dbc3.expire_snapshots'
> E CAUSED BY: TableLoadingException: Error opening Iceberg table
> 'test_expire_snapshots_e488dbc3.expire_snapshots'
> E CAUSED BY: InconsistentMetadataFetchException: Catalog object
> TCatalogObject(type:TABLE, catalog_version:7309,
> table:TTable(db_name:test_expire_snapshots_e488dbc3,
> tbl_name:expire_snapshots)) changed version between accesses.
> {noformat}
> The error might be due to not detecting all self-events correctly, so the
> IcebergTable gets updated on the CatalogD side.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
