Nándor Kollár created IMPALA-15736:
--------------------------------------
Summary: ALTER TABLE DROP PARTITION on Iceberg tables can drop
concurrently written files
Key: IMPALA-15736
URL: https://issues.apache.org/jira/browse/IMPALA-15736
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Nándor Kollár
If ALTER TABLE DROP PARTITION selects every file of an Iceberg table, the
analysis replaces it with a truncate: the file paths are compared with the
total number of files in the snapshot summary, and then catalogd commits
DeleteFiles.deleteFromRowFilter(alwaysTrue()) instead of deleting the selected
files. The decision is made on the snapshot the analysis saw, but the filter is
evaluated on the table when catalogd commits. Files committed in between, by
another Impala query or by another engine, are dropped too, even if they are in
partitions that the statement doesn't select. Nothing validates that the table
didn't change: DeleteFiles has no validateFromSnapshot(), and when the commit
conflicts, Iceberg refreshes the table and applies alwaysTrue() again to the
new state (BaseTransaction retries up to commit.retry.num-retries times).
Example: table t is partitioned by p and has the partitions p=1 and p=2.
1. ALTER TABLE t DROP PARTITION (p IN (1, 2)) is analyzed: both files are
selected, so the drop becomes a truncate.
2. A concurrent INSERT INTO t VALUES (3, ...) commits a file in the new
partition p=3.
3. catalogd commits the truncate: the table is empty, and the row of p=3 is
lost although the INSERT succeeded.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
