Hello Gabor Kaszab, wangsheng, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16851
to look at the new patch set (#2).
Change subject: IMPALA-10358: Correct Iceberg type mappings
......................................................................
IMPALA-10358: Correct Iceberg type mappings
The Iceberg format spec defines what types to use for different file
formats, e.g.: https://iceberg.apache.org/spec/#parquet
Impala should follow the specification, so this patch
* annotates strings with UTF8 in Parquet metadata
* removes fixed(L) <-> CHAR(L) mapping
* forbids INSERTs when the Iceberg schema has a TIMESTAMPTZ column
This patch also refactors the type/schema conversions as
Impala => Iceberg conversions were duplicated in
IcebergCatalogOpExecutor and IcebergUtil. I introduced the class
'IcebergSchemaConverter' to contain the code for conversions.
Testing:
* added test to check CHAR and VARCHAR types are not allowed
* test that INSERTs are not allowed when the table has TIMESTMAPTZ
* added test to check that strings are annotated with UTF8
Change-Id: I652565f82708824f5cf7497139153b06f116ccd3
---
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/exec/parquet/hdfs-parquet-table-writer.h
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
A fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
M tests/query_test/test_iceberg.py
14 files changed, 303 insertions(+), 273 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/16851/2
--
To view, visit http://gerrit.cloudera.org:8080/16851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I652565f82708824f5cf7497139153b06f116ccd3
Gerrit-Change-Number: 16851
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Gabor Kaszab <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
Gerrit-Reviewer: wangsheng <[email protected]>
