[
https://issues.apache.org/jira/browse/IMPALA-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808354#comment-17808354
]
ASF subversion and git services commented on IMPALA-11710:
----------------------------------------------------------
Commit 47e2cc3bb9b982e6b9f30535fa7374f2da1b8dd3 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=47e2cc3bb ]
IMPALA-11710: Table properties are not updated in Iceberg metadata files
Impala has some troubles with Iceberg tables that don't specify
'external.table.purge'='true'. Schema changes like ADD COLUMN is not
working, also table properties that have been set by an ALTER TABLE
SET TBLPROPERTIES statement can be reset by subsequent INSERT
statements.
There was a bug in CatalogOpExecutor.alterIcebergTables(). Its return
value determines whether we need to also update the table definition in
HMS, or it was already done by the Iceberg library. In fact, the HMS
table definition is always updated by the Iceberg library if the table
is handled by the HiveCatalog. In every other case we need to update
the HMS table definition ourselves (unless the change won't affect
HMS).
The issue was that CatalogOpExecutor.alterIcebergTables() returned true
(which means we need to update HMS) in case of Iceberg tables that
didn't specify 'external.table.purge'='true'. This was a problem,
because Iceberg already modified the HMS table and set
'metadata_location' to a new metadata file. But then Impala came and
modified some properties of the HMS table definition, but also reset
'metadata_location' to the original value. Therefore subsequent
operations/refreshes used the earlier state of the Iceberg table and
they reset the modifications.
Testing:
* added e2e tests for Iceberg tables in different catalogs with
'external.table.merge'='false'
Change-Id: I2a82d022534e1e212d542fbd7916ae033c381c20
Reviewed-on: http://gerrit.cloudera.org:8080/20907
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Table properties are not updated in Iceberg metadata files
> ----------------------------------------------------------
>
> Key: IMPALA-11710
> URL: https://issues.apache.org/jira/browse/IMPALA-11710
> Project: IMPALA
> Issue Type: Bug
> Reporter: Noemi Pap-Takacs
> Assignee: Zoltán Borók-Nagy
> Priority: Critical
> Labels: impala-iceberg
>
> This issue occurs in true external Hive Catalog tables.
> Iceberg stores the default file format in a table property called
> 'write.format.default'. HMS also stores this value loaded from the Iceberg
> metadata json file.
> However, when this table property is altered through Impala, it is only
> changed in HMS, but does not update the Iceberg snapshot. When the table data
> is reloaded from Iceberg metadata, the old value will appear in HMS and the
> change is lost.
> This bug does not affect table properties that are not stored in Iceberg,
> because they will not be reloaded.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
