Hello Zoltan Borok-Nagy, Daniel Vanko, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24238
to look at the new patch set (#6).
Change subject: IMPALA-14589: [Part 2] Add comprehensive type support for
Iceberg initial-default values
......................................................................
IMPALA-14589: [Part 2] Add comprehensive type support for Iceberg
initial-default values
This patch adds support for initial-default values across all primitive
types in Iceberg V3 tables, addressing limitations where only numeric
types were previously handled correctly.
Background:
Iceberg stores default values in metadata.json, but the Iceberg Literal
API returns internal representations that differ from what Impala's
TextConverter expects for parsing. For example, DATE is stored as an
ISO-8601 string in JSON but Literal.value() returns an Integer
(days since epoch), and TIMESTAMP returns microseconds as a Long.
Testing:
- Added iceberg_v3_all_types test table with initial-default values for
all supported primitive types: BOOLEAN, INT, BIGINT, FLOAT, DOUBLE,
DECIMAL, STRING, DATE, TIMESTAMP, TIMESTAMPTZ and BINARY
- Added positive test cases for supported types
- Added negative test case for BINARY type rejection
- Updated testdata/data/README with table documentation
Change-Id: I1c591d5dbf305176dd802c6f5f811c0de98aac0c
---
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java
M testdata/data/README
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_all_types/data/00000-6-fb62b831-267c-4614-b85c-829318a8ff67-0-00001.parquet
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_all_types/data/00000-7-3180aeb0-e342-4d28-87ad-87847496148c-0-00001.parquet
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_all_types/metadata/93172531-868e-45f3-bc08-f0ba609959c1-m0.avro
R
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_all_types/metadata/ee2a974f-f16a-4796-8f47-adfdca33dd35-m0.avro
C
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_all_types/metadata/snap-2190628750428777669-1-93172531-868e-45f3-bc08-f0ba609959c1.avro
C
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_all_types/metadata/snap-4685659321801244137-1-ee2a974f-f16a-4796-8f47-adfdca33dd35.avro
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_all_types/metadata/v1.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_all_types/metadata/v2.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_all_types/metadata/v3.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_all_types/metadata/v4.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_all_types/metadata/version-hint.txt
R
testdata/data/iceberg_test/iceberg_v3/test_complex_default/data/00000-1-3cf46297-b775-416f-958e-3e3e54ef8015-0-00001.parquet
D
testdata/data/iceberg_test/iceberg_v3/test_complex_default/data/00000-1-e5d7e769-5fc4-461d-a35b-f15a3779c122-0-00001.parquet
A
testdata/data/iceberg_test/iceberg_v3/test_complex_default/data/00000-2-e8bd1d55-a8ab-41a1-a581-f8c8fb49ef11-0-00001.parquet
R
testdata/data/iceberg_test/iceberg_v3/test_complex_default/metadata/e903850b-4d6b-45f0-b707-b864dd3308a6-m0.avro
A
testdata/data/iceberg_test/iceberg_v3/test_complex_default/metadata/eda9c488-3983-4aab-b487-aaab97071435-m0.avro
C
testdata/data/iceberg_test/iceberg_v3/test_complex_default/metadata/snap-1631941062721562552-1-eda9c488-3983-4aab-b487-aaab97071435.avro
D
testdata/data/iceberg_test/iceberg_v3/test_complex_default/metadata/snap-2556729165577948823-1-45121802-1f9f-49ed-b088-fbbe0c63a810.avro
R
testdata/data/iceberg_test/iceberg_v3/test_complex_default/metadata/snap-8272972260788580730-1-e903850b-4d6b-45f0-b707-b864dd3308a6.avro
M
testdata/data/iceberg_test/iceberg_v3/test_complex_default/metadata/v1.metadata.json
M
testdata/data/iceberg_test/iceberg_v3/test_complex_default/metadata/v2.metadata.json
M
testdata/data/iceberg_test/iceberg_v3/test_complex_default/metadata/v3.metadata.json
M
testdata/data/iceberg_test/iceberg_v3/test_complex_default/metadata/v4.metadata.json
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-default-values.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-negative.test
M tests/query_test/test_iceberg.py
29 files changed, 393 insertions(+), 180 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/24238/6
--
To view, visit http://gerrit.cloudera.org:8080/24238
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1c591d5dbf305176dd802c6f5f811c0de98aac0c
Gerrit-Change-Number: 24238
Gerrit-PatchSet: 6
Gerrit-Owner: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Daniel Vanko <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
