Tamas Mate has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/20010 )
Change subject: IMPALA-11996: Scanner change for Iceberg metadata querying ...................................................................... IMPALA-11996: Scanner change for Iceberg metadata querying This commit adds a scan node for querying Iceberg metadata tables. The scan node creates a Java scanner object that creates and scans the metadata table. The scanner uses the Iceberg API to scan the table after that the scan node fetches the rows one by one and materialises them into RowBatches. The Iceberg row reader on the backend does the translation between Iceberg and Impala types. There is only one fragment created to query the Iceberg metadata table which is supposed to be executed on the coordinator node that already has the Iceberg table loaded. This way there is no need for further table loading on the executor side. This change will not cover struct column types, these slots are set to NULL, it will be done in IMPALA-12205. Testing: - Added e2e tests for querying metadata tables - Updated planner tests Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 --- M be/CMakeLists.txt M be/src/exec/CMakeLists.txt M be/src/exec/exec-node.cc M be/src/exec/hbase/hbase-table-scanner.cc A be/src/exec/iceberg-metadata/CMakeLists.txt A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h A be/src/exec/iceberg-metadata/iceberg-row-reader.cc A be/src/exec/iceberg-metadata/iceberg-row-reader.h M be/src/scheduling/scheduler.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impalad-main.cc M be/src/util/jni-util.cc M be/src/util/jni-util.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergMetadataTableRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java A testdata/data/iceberg_test/iceberg_test_metadata/data/3d481ed88b2941f0-ea33816200000000_1109948289_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/944a2355e618932f-18f086b600000000_1283312202_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/d64ba652b36f2d76-d075207600000000_883987622_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-1f43b217940cc094-fedf515600000000_248998721_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-bc4b163507d1da6f-a691cf6100000002_511900219_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/metadata/38e5a1bd-5b7f-4eae-9362-16a2de3c575d-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/7c952273-c5d6-43d2-bb1b-72fbe1555f25-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/88a39285-529f-41a4-bd69-6d2560fac64e-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/ef78f8eb-b2b5-41df-a2a0-7944ac3c38da-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-7858675898458780516-1-ef78f8eb-b2b5-41df-a2a0-7944ac3c38da.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8283026816932323050-1-38e5a1bd-5b7f-4eae-9362-16a2de3c575d.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8491702501245661704-1-88a39285-529f-41a4-bd69-6d2560fac64e.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-9046920472784493998-1-7c952273-c5d6-43d2-bb1b-72fbe1555f25.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v4.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v5.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/version-hint.txt M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-metadata-table-scan.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test 49 files changed, 1,730 insertions(+), 157 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20010/6 -- To view, visit http://gerrit.cloudera.org:8080/20010 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 Gerrit-Change-Number: 20010 Gerrit-PatchSet: 6 Gerrit-Owner: Tamas Mate <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Gabor Kaszab <[email protected]> Gerrit-Reviewer: Gergely Fürnstáhl <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tamas Mate <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
