Zoltan Borok-Nagy has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/21950 )
Change subject: IMPALA-13463: Impala should ignore case of Iceberg schema
elements
......................................................................
IMPALA-13463: Impala should ignore case of Iceberg schema elements
Schema is case insensitive in Impala. Via Spark it's possible to create
schema elements with upper/lower case letters and store them in the
metadata JSON files of Iceberg, e.g.:
"schemas" : [ {
"type" : "struct",
"schema-id" : 0,
"fields" : [ {
"id" : 1,
"name" : "ID",
"required" : false,
"type" : "string"
}, {
"id" : 2,
"name" : "OWNERID",
"required" : false,
"type" : "string"
} ]
} ],
This can cause problems in Impala during predicate pushdown, as we can
get a ValidationException from the Iceberg library (as Impala pushes
down predicates with lower case column names, while Iceberg sees upper
case names).
With this patch Impala invokes Scan.caseSensitive(boolean caseSensitive)
on the TableScan object to set case insensitivity.
Testing:
* added e2e test
Change-Id: Iedaf152d8a0c02a124c3dcf8acb59b4ba4e81cf4
Reviewed-on: http://gerrit.cloudera.org:8080/21950
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Wenzhe Zhou <[email protected]>
Reviewed-by: Daniel Becker <[email protected]>
---
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A
testdata/data/iceberg_test/iceberg_column_case_sensitivity_issue/data/id_bucket=3/5b4ef6d2e91c0129-f018b1d800000000_872469098_data.0.parq
A
testdata/data/iceberg_test/iceberg_column_case_sensitivity_issue/data/id_bucket=7/504c5f5ae97c4c65-c9fce43a00000000_1852333400_data.0.parq
A
testdata/data/iceberg_test/iceberg_column_case_sensitivity_issue/metadata/1a457d69-768a-4bfd-8da5-c080d3b88e50-m0.avro
A
testdata/data/iceberg_test/iceberg_column_case_sensitivity_issue/metadata/96461a99-3b56-4573-ab6d-8b8ba3fbcae2-m0.avro
A
testdata/data/iceberg_test/iceberg_column_case_sensitivity_issue/metadata/snap-1855055649619147667-1-96461a99-3b56-4573-ab6d-8b8ba3fbcae2.avro
A
testdata/data/iceberg_test/iceberg_column_case_sensitivity_issue/metadata/snap-7743982156242154468-1-1a457d69-768a-4bfd-8da5-c080d3b88e50.avro
A
testdata/data/iceberg_test/iceberg_column_case_sensitivity_issue/metadata/v3.metadata.json
A
testdata/data/iceberg_test/iceberg_column_case_sensitivity_issue/metadata/version-hint.text
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-column-case-sensitivity-issue.test
M tests/query_test/test_iceberg.py
11 files changed, 161 insertions(+), 2 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Wenzhe Zhou: Looks good to me, but someone else must approve
Daniel Becker: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/21950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iedaf152d8a0c02a124c3dcf8acb59b4ba4e81cf4
Gerrit-Change-Number: 21950
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Daniel Becker <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Wenzhe Zhou <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
