Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/25041
to look at the new patch set (#2).
Change subject: IMPALA-15212: Iceberg UUID ORC read support
......................................................................
IMPALA-15212: Iceberg UUID ORC read support
Iceberg stores UUID columns in ORC files as BINARY with the
iceberg.binary-type=UUID attribute. Impala now reads them into
TYPE_UUID slots, so Iceberg tables with UUID columns can be scanned
when they contain ORC data files. This works for UUIDs at the top
level and inside STRUCT, ARRAY and MAP columns. Avro data files with
UUID columns are still rejected.
The ORC schema resolver maps BINARY to UUID table columns. A new
OrcUuidColumnReader copies the 16 bytes of each value from the ORC
batch into the inline slot. A value of any other length, or a
dictionary-encoded batch, fails the scan as a corrupt file.
UUID predicates (=, IN, range, IS NULL and IS NOT NULL) are pushed to
the ORC reader as STRING search arguments on the 16 raw bytes.
ORC BINARY columns have no min/max statistics, so the reader can only
skip data by null counts and, for = and IN, by bloom filters, which
hash the same raw bytes. Range predicates are still evaluated on the
pushed search arguments (16-byte unsigned byte order) but cannot prune
stripes for the same reason. IN-list runtime filters do not
support UUID.
The planner's memory reservation estimate for ORC async reads now
counts the LENGTH stream of UUID columns.
Testing:
- Regenerated the iceberg_uuid_test_orc fixture as a single ORC data
file with a bloom filter on uuid_col, using the Iceberg Java API
(version 1.10.1).
- e2e tests on iceberg_uuid_test_orc: SELECT, =, !=, IN, NOT IN,
range, IS NULL and IS NOT NULL predicates, ORDER BY, aggregates,
CAST, a JOIN with the Parquet fixture, EXPLAIN checks that UUID
predicates are ORC statistics predicates, and a check that an
equality predicate on an absent value skips the data through the
bloom filter.
- Trino interop tests: Impala reads a table with both Parquet and ORC
data files, and a Trino-written ORC table with UUIDs at the top
level and in STRUCT, ARRAY and MAP, with values on both sides of the
sign bit, including a JOIN with a Parquet table.
Change-Id: Id736a9ea852892b610281df9105c82e4a4ce42da
Assisted-by: Claude Opus 5.5 (Cursor)
---
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/orc/hdfs-orc-scanner.h
M be/src/exec/orc/orc-column-readers.cc
M be/src/exec/orc/orc-column-readers.h
M be/src/exec/orc/orc-metadata-utils.cc
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/data/README
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-2e0fbfe3-d923-4a27-b5d4-f941a82fb1d1-1-00001.orc
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-896e60f8-2a60-47e3-8187-8f8bed145b7f-1-00001.orc
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-b9172fb6-1d4c-46e0-817c-b3d6377ccf31-1-00001.orc
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-d7da9c66-c7e5-45e0-a69a-0c2eb82e96c0-1-00001.orc
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-eef53fbc-bade-487e-b510-537354cf46e0-1-00001.orc
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/data/00000-0-ffed3f33-e038-42ff-8c17-8e03204fd3e0-1-00001.orc
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/22fa5c4b-83a3-46b2-bafa-3553189862bd-m0.avro
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/4abe1f4e-1220-4b05-a5e6-ebfc8fb92df3-m0.avro
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/547a17ba-a0d2-4921-956b-e06fbdadb60b-m0.avro
R
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/92eaf5b5-cb0d-4589-a10d-a5622ead529f-m0.avro
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/d570e583-e57a-4879-8a99-052c9a85d706-m0.avro
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-5116163803842658326-1-4abe1f4e-1220-4b05-a5e6-ebfc8fb92df3.avro
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-5748673676392974692-1-75312a76-0d22-45dc-80c7-41c2cb712f98.avro
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-6680088367048559051-1-22fa5c4b-83a3-46b2-bafa-3553189862bd.avro
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-7102485323114575692-1-547a17ba-a0d2-4921-956b-e06fbdadb60b.avro
R
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/snap-7535938409123090183-1-92eaf5b5-cb0d-4589-a10d-a5622ead529f.avro
M
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v1.metadata.json
M
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v2.metadata.json
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v3.metadata.json
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v4.metadata.json
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v5.metadata.json
D
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/v6.metadata.json
M
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_orc/metadata/version-hint.text
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-uuid.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-uuid-type.test
M tests/custom_cluster/test_iceberg_trino_interop.py
34 files changed, 383 insertions(+), 29 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/25041/2
--
To view, visit http://gerrit.cloudera.org:8080/25041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id736a9ea852892b610281df9105c82e4a4ce42da
Gerrit-Change-Number: 25041
Gerrit-PatchSet: 2
Gerrit-Owner: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
