Noémi Pap-Takács created IMPALA-15483:
-----------------------------------------
Summary: DROP PARTITION leaves delete files in the Iceberg table
Key: IMPALA-15483
URL: https://issues.apache.org/jira/browse/IMPALA-15483
Project: IMPALA
Issue Type: Bug
Reporter: Noémi Pap-Takács
ALTER TABLE <t> DROP PARTITION (<pred>) removes the partition's data files from
the table metadata but keeps its delete files — position deletes, equality
deletes and deletion vectors — referenced by the new snapshot.
Repro (Iceberg 1.11.0):
{code:java}
CREATE TABLE t (i INT, s STRING) PARTITIONED BY SPEC(i)
STORED BY ICEBERG TBLPROPERTIES ('format-version'='2');
INSERT INTO t VALUES (1,'a'),(1,'b');
INSERT INTO t VALUES (2,'c');
DELETE FROM t WHERE s = 'a'; -- position delete in partition i=1
ALTER TABLE t DROP PARTITION (i = 1);
SELECT content, `partition`.i, file_path FROM t.`files`;
Actual: a content=1, i=1 row remains (.../data/i=1/delete-*.parq).
Expected: no files of partition i=1.{code}
Root cause: AlterTableDropPartitionStmt.analyzeIceberg collects the data files
and FileScanTask.deletes() into the path list, and
IcebergCatalogOpExecutor.alterTableDropPartition feeds each path to
DeleteFiles.deleteFile(String). In Iceberg,
MergingSnapshotProducer.delete(CharSequence) registers the path only with the
data-file ManifestFilterManager; delete-file manifests are filtered solely
through the typed delete(DeleteFile) overload. The delete-file paths are
therefore silently dropped, and nothing reports it because validateFilesExist()
is not enabled.
Impact: query results are unaffected — with its data files gone the stale
delete file matches nothing, and later inserts carry higher data sequence
numbers. However the entries persist in the manifests and in the
files/delete_files metadata tables, total-delete-files stays inflated, and the
files remain referenced, so orphan-file cleanup cannot collect them. Only
partial drops are affected: a drop that selects every file takes the
is_truncate path (deleteFromRowFilter(alwaysTrue)), which is clean.
Fix direction: delete typed — re-plan the partition at commit time and call
deleteFile(DataFile) / deleteFile(DeleteFile), or split the thrift paths field
into data-file and delete-file paths.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
