Hello Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24662
to look at the new patch set (#5).
Change subject: IMPALA-15006: SHOW FILES does not list deletion vectors for
Iceberg V3 tables
......................................................................
IMPALA-15006: SHOW FILES does not list deletion vectors for Iceberg V3 tables
Iceberg V3 tables store deletion vectors (DVs) in PUFFIN format files.
Previously, SHOW FILES did not include these files because
IcebergContentFileStore.getAllFiles() only returned data files and
position/equality delete files, omitting DVs from the dataFileToDV_ map.
This patch:
- Introduces TIcebergDeleteFiles thrift struct to unify serialization of
all delete file types (position deletes, equality deletes, and DVs).
- Adds DV puffin files to the SHOW FILES output, both for unfiltered
listings and partition-filtered/time-travel queries.
- Refactors SHOW FILES to use the FeShowFileStmtSupport interface,
delegating table-type-specific partition filter validation and file
listing to the table type (Iceberg, HDFS, Paimon). This removes
Iceberg-specific logic from ShowFilesStmt and makes it generic.
- Moves transformPartitionConjuncts into FeFsTable.Utils for reuse.
- Removes the now-redundant paimon/FeShowFileStmtSupport.java in favor
of the shared catalog-level interface.
- Adds "Delete Type" and "Record Count" columns to SHOW FILES output
for Iceberg tables to distinguish data files, position deletes,
equality deletes, and deletion vectors.
Testing:
- Updated AnalyzeDDLTest to match new error message paths.
- Updated .test files for the additional output columns.
Change-Id: Ica86830da6cc1eba335159612b73a905a339b064
---
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java
M
fe/src/main/java/org/apache/impala/analysis/IcebergPartitionExpressionRewriter.java
M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
A fe/src/main/java/org/apache/impala/catalog/FeShowFileStmtSupport.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
M fe/src/main/java/org/apache/impala/catalog/paimon/FePaimonTable.java
D fe/src/main/java/org/apache/impala/catalog/paimon/FeShowFileStmtSupport.java
M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonUtil.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-catalogs.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-delete-partitioned.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-drop-partition.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-optimize.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-show-files-partition.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-update-partitions.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-delete-partitioned.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-update-partitions.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-write-many-files-stress.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-write-many-files.test
M testdata/workloads/functional-query/queries/QueryTest/paimon-show-stats.test
31 files changed, 1,400 insertions(+), 1,109 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/24662/5
--
To view, visit http://gerrit.cloudera.org:8080/24662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ica86830da6cc1eba335159612b73a905a339b064
Gerrit-Change-Number: 24662
Gerrit-PatchSet: 5
Gerrit-Owner: Nandor Kollar <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Nandor Kollar <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
