[
https://issues.apache.org/jira/browse/IMPALA-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653609#comment-17653609
]
Zoltán Borók-Nagy commented on IMPALA-10166:
--------------------------------------------
[~davidxdh] , I don't think we should add support for such statements. Data
files should be added to the Iceberg tables via Iceberg API, not just by
placing them to the table directory.
I guess the use case here is to write the data files via some tool and place
them into an Iceberg table's partition directory, then invoke RECOVER
PARTITIONS to add the new data files to the table.
The problem with such data files is that they might not conform to Iceberg
specifications, e.g. they might have missing field ids. Because of this even
our LOAD DATA INPATH statement rewrites the data files: IMPALA-11339.
With Iceberg's advanced partitioning such as partition transforms it's not
trivial to come up with the proper partition names and values for external
tools. Also, the table might place its files outside the table location, e.g.
see
[ObjectStoreLocationProvider|https://github.com/apache/iceberg/blob/adecd8aa6f1e57974f115a2e33b4288eb044a4bd/core/src/main/java/org/apache/iceberg/LocationProviders.java#L106:16]
Deleted data files belonging to old snapshots might still remain under the
table location. How are we going to differentiate between such files and the
new files?
All in all, I think RECOVER PARTITION statement is problematic and not really
compatible with Iceberg tables.
> ALTER TABLE for Iceberg tables
> ------------------------------
>
> Key: IMPALA-10166
> URL: https://issues.apache.org/jira/browse/IMPALA-10166
> Project: IMPALA
> Issue Type: New Feature
> Components: Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Sheng Wang
> Priority: Major
> Labels: impala-iceberg
> Fix For: Impala 4.1.0
>
>
> Add support for ALTER TABLE operations for Iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
