Nandor Kollar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24662
Change subject: IMPALA-15006: SHOW FILES does not list deletion vectors for Iceberg V3 tables ...................................................................... IMPALA-15006: SHOW FILES does not list deletion vectors for Iceberg V3 tables Iceberg V3 tables store deletion vectors (DVs) in PUFFIN format files. Previously, SHOW FILES did not include these files because IcebergContentFileStore.getAllFiles() only returned data files and position/equality delete files, omitting DVs from the dataFileToDV_ map. This patch: - Introduces TIcebergDeleteFiles thrift struct to unify serialization of all delete file types (position deletes, equality deletes, and DVs). - Adds DV puffin files to the SHOW FILES output, both for unfiltered listings and partition-filtered/time-travel queries. - Refactors SHOW FILES to use the FeShowFileStmtSupport interface, delegating table-type-specific partition filter validation and file listing to the table type (Iceberg, HDFS, Paimon). This removes Iceberg-specific logic from ShowFilesStmt and makes it generic. - Moves transformPartitionConjuncts into FeFsTable.Utils for reuse. - Removes the now-redundant paimon/FeShowFileStmtSupport.java in favor of the shared catalog-level interface. - Adds "Delete Type" and "Record Count" columns to SHOW FILES output for Iceberg tables to distinguish data files, position deletes, equality deletes, and deletion vectors. Testing: - Updated AnalyzeDDLTest to match new error message paths. - Updated .test files for the additional output columns. Change-Id: Ica86830da6cc1eba335159612b73a905a339b064 --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionExpressionRewriter.java M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java A fe/src/main/java/org/apache/impala/catalog/FeShowFileStmtSupport.java M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java M fe/src/main/java/org/apache/impala/catalog/paimon/FePaimonTable.java D fe/src/main/java/org/apache/impala/catalog/paimon/FeShowFileStmtSupport.java M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonUtil.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-catalogs.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-delete-partitioned.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-drop-partition.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-optimize.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-show-files-partition.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-update-partitions.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-delete-partitioned.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-update-partitions.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-write-many-files-stress.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-write-many-files.test M testdata/workloads/functional-query/queries/QueryTest/paimon-show-stats.test 29 files changed, 1,393 insertions(+), 1,048 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/24662/1 -- To view, visit http://gerrit.cloudera.org:8080/24662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ica86830da6cc1eba335159612b73a905a339b064 Gerrit-Change-Number: 24662 Gerrit-PatchSet: 1 Gerrit-Owner: Nandor Kollar <[email protected]>
