Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19776 )
Change subject: IMPALA-11877: (part 1) Add support for DELETE statements for UNPARTITIONED Iceberg tables ...................................................................... IMPALA-11877: (part 1) Add support for DELETE statements for UNPARTITIONED Iceberg tables This patch adds support for DELETE statements on unpartitioned Iceberg tables. Impala uses the 'merge-on-read' mode with position delete files. The patch reuses the existing IcebergPositionDeleteTable as the target table of the DELETE statements, because this table already has the same schema as position delete files, even with correct Iceberg field IDs. The patch basically rewrites DELETE statements to INSERT statements, e.g.: from: DELETE FROM ice_t WHERE id = 42; to: INSERT INTO ice_t-POSITION-DELETE SELECT INPUT__FILE__NAME, FILE__POSITION FROM ice_t WHERE id = 42; Position delete files need to be ordered by (file_path, pos), so we add an extra SORT node before the table sink operator. In the backend the patch adds a new table sink operator, the IcebergDeleteSink. It writes the incoming rows (file_path, position) to delete files. It reuses a lot of code from HdfsTableSink, so this patch moves the common code to the new common base class: TableSinkBase. The coordinator then collects the written delete files and invokes UpdateCatalog to finalize the DELETE statement. The Catalog then uses Iceberg APIs to create a new snapshot with the created delete files. It also validates that there was no conflicting data files written since the operation started. Testing: * added planer test * e2e tests * interop test between Impala and Hive Change-Id: Ic933b2295abe54b46d2a736961219988ff42915b Reviewed-on: http://gerrit.cloudera.org:8080/19776 Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Gabor Kaszab <[email protected]> --- M be/src/exec/CMakeLists.txt M be/src/exec/data-sink.cc M be/src/exec/file-metadata-utils.cc M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/exec/hdfs-table-writer.cc M be/src/exec/hdfs-table-writer.h A be/src/exec/iceberg-delete-sink.cc A be/src/exec/iceberg-delete-sink.h M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/hdfs-parquet-table-writer.h A be/src/exec/table-sink-base.cc A be/src/exec/table-sink-base.h M be/src/runtime/coordinator.cc M be/src/runtime/descriptors.cc M be/src/service/client-request-state.cc M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M common/thrift/DataSinks.thrift M common/thrift/Query.thrift M common/thrift/Types.thrift M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/VirtualColumn.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java A fe/src/main/java/org/apache/impala/planner/IcebergDeleteSink.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeModifyStmtsTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-delete.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-delete-complex.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-delete.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test M tests/query_test/test_iceberg.py 44 files changed, 1,858 insertions(+), 465 deletions(-) Approvals: Impala Public Jenkins: Verified Gabor Kaszab: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/19776 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic933b2295abe54b46d2a736961219988ff42915b Gerrit-Change-Number: 19776 Gerrit-PatchSet: 13 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Andrew Sherman <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Gabor Kaszab <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tamas Mate <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
