Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24505
to look at the new patch set (#6).
Change subject: IMPALA-15200: Add Parquet UUID read support for Iceberg tables
......................................................................
IMPALA-15200: Add Parquet UUID read support for Iceberg tables
This patch enables Impala to read UUID columns from Iceberg tables
stored in Parquet format, building on IMPALA-15101's inline TYPE_UUID
storage. Parquet stores UUID as FIXED_LEN_BYTE_ARRAY(16) with UUID
logical type metadata. The scanner reads the 16 raw bytes and writes
them directly into the inline TYPE_UUID tuple slot.
Parquet scan/decode path:
- ScalarColumnReader specializations for PLAIN and dictionary-encoded
UUID
- Column stats decode, metadata validation, and data converter wiring
- Codegen StringVal layout for TYPE_UUID
- Text converter for Iceberg V3 initial-default backfill
- Scalar expr evaluator slot materialization
- Display/formatting (raw-value, HS2, query results) for SELECT
assertions
- FE planner/catalog: ColumnStats (fixed-length HMS stats)
Testing:
- Query tests: DESCRIBE, SELECT with UUID values, SHOW CREATE,
Iceberg V3 initial-default, negative ORC/Avro read, and negative
predicate/cast/aggregate tests
- Test fixtures added in parquet, avro, and orc formats
Change-Id: I4157c002e80677d27d8fd060c7bfa07b95d7c78f
Assisted-by: Composer 2.5
---
M be/src/codegen/codegen-anyval-ir.cc
M be/src/codegen/codegen-anyval.cc
M be/src/codegen/gen_ir_descriptions.py
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-column-stats.cc
M be/src/exec/parquet/parquet-column-stats.h
M be/src/exec/parquet/parquet-common.h
M be/src/exec/parquet/parquet-data-converter.h
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/text-converter.inline.h
M be/src/exprs/scalar-expr-evaluator-ir.cc
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/runtime/raw-value-ir.cc
M be/src/runtime/raw-value.cc
M be/src/runtime/raw-value.inline.h
M be/src/service/fe-support.cc
M be/src/service/hs2-util.cc
M be/src/service/query-result-set.cc
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
M testdata/data/README
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/data/00000-0-4506acbe-4b67-470a-a55e-b10babd0d936-1-00001.parquet
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/data/00000-0-6a690319-54bc-488d-ad09-72e0728adcf1-1-00001.parquet
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/data/00000-0-87520c0c-721e-47c2-a13e-c532a6d5506d-1-00001.parquet
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/data/00000-0-a1b65880-dbb4-4723-8a62-38ab685f57a5-1-00001.parquet
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/data/00000-0-c8b87271-588a-450b-b893-fe3503b9e0e0-1-00001.parquet
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/5dbfe621-ea8b-42ef-bf71-22924ebe9a4c-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/6bab8911-ac05-48c9-a5f5-840b2f5b2158-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/a524d5c5-eb17-492a-b20a-aef9dfaf77eb-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/a8a0fea1-9d07-4af1-a580-c28fe74d58a9-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/f3bde471-ff6e-433a-b184-71564eac14f2-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/snap-1854301121202970963-1-6bab8911-ac05-48c9-a5f5-840b2f5b2158.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/snap-4744360932271692267-1-a8a0fea1-9d07-4af1-a580-c28fe74d58a9.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/snap-4979207191920000664-1-5dbfe621-ea8b-42ef-bf71-22924ebe9a4c.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/snap-5570936655938254077-1-f3bde471-ff6e-433a-b184-71564eac14f2.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/snap-6711875576164514304-1-a524d5c5-eb17-492a-b20a-aef9dfaf77eb.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/v1.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/v2.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/v3.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/v4.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/v5.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/v6.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/v7.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test/metadata/version-hint.text
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/data/00000-0-36a10355-4da3-48cc-baa7-78a818130e15-1-00001.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/data/00000-0-4e7c2ed8-d3ce-484c-8bc5-d0bd095c46d1-1-00001.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/data/00000-0-e1248e53-2a33-434c-ba9e-4b0cde09f23a-1-00001.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/data/00000-0-e290bcce-35c3-48d5-b592-da066823f124-1-00001.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/data/00000-0-e6f6ea8e-0241-44c6-b214-1aa107927521-1-00001.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/2b500dae-2f17-4139-a3ab-749bf7d4d2b4-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/2f0ca499-85fc-4cf7-a4d7-abf2f4a18a46-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/c0ce300a-9874-4199-aa3a-abcae5b02ef7-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/d1bbde7d-1667-454f-b354-5661c08c8e94-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/d2878e39-671e-434e-a1cf-e6f49bb218b5-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/snap-1257189622857586261-1-d2878e39-671e-434e-a1cf-e6f49bb218b5.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/snap-5325065939761959235-1-2b500dae-2f17-4139-a3ab-749bf7d4d2b4.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/snap-5960700658108006852-1-c0ce300a-9874-4199-aa3a-abcae5b02ef7.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/snap-6779792615814615855-1-d1bbde7d-1667-454f-b354-5661c08c8e94.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/snap-9054099040941058343-1-2f0ca499-85fc-4cf7-a4d7-abf2f4a18a46.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/v1.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/v2.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/v3.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/v4.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/v5.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/v6.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_avro/metadata/version-hint.text
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-2e0fbfe3-d923-4a27-b5d4-f941a82fb1d1-1-00001.orc
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-896e60f8-2a60-47e3-8187-8f8bed145b7f-1-00001.orc
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-b9172fb6-1d4c-46e0-817c-b3d6377ccf31-1-00001.orc
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-d7da9c66-c7e5-45e0-a69a-0c2eb82e96c0-1-00001.orc
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-eef53fbc-bade-487e-b510-537354cf46e0-1-00001.orc
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/22fa5c4b-83a3-46b2-bafa-3553189862bd-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/4abe1f4e-1220-4b05-a5e6-ebfc8fb92df3-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/547a17ba-a0d2-4921-956b-e06fbdadb60b-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/75312a76-0d22-45dc-80c7-41c2cb712f98-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/d570e583-e57a-4879-8a99-052c9a85d706-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-5116163803842658326-1-4abe1f4e-1220-4b05-a5e6-ebfc8fb92df3.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-5347252638781720483-1-d570e583-e57a-4879-8a99-052c9a85d706.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-5748673676392974692-1-75312a76-0d22-45dc-80c7-41c2cb712f98.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-6680088367048559051-1-22fa5c4b-83a3-46b2-bafa-3553189862bd.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-7102485323114575692-1-547a17ba-a0d2-4921-956b-e06fbdadb60b.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v1.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v2.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v3.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v4.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v5.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v6.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/version-hint.text
A testdata/workloads/functional-query/queries/QueryTest/iceberg-uuid-type.test
M tests/query_test/test_iceberg.py
95 files changed, 489 insertions(+), 39 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/24505/6
--
To view, visit http://gerrit.cloudera.org:8080/24505
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4157c002e80677d27d8fd060c7bfa07b95d7c78f
Gerrit-Change-Number: 24505
Gerrit-PatchSet: 6
Gerrit-Owner: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
