Hello Tamas Mate, Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17398
to look at the new patch set (#2).
Change subject: IMPALA-10485: Support Iceberg field-id based column resolution
in the ORC scanner
......................................................................
IMPALA-10485: Support Iceberg field-id based column resolution in the ORC
scanner
Currently the ORC scanner only supports position-based column
resolution. This patch adds Iceberg field-id based column resolution
which will be the default for Iceberg tables. It is needed to support
schema evolution in the future, i.e. ALTER TABLE DROP/RENAME COLUMNS.
(The Parquet scanner already supports Iceberg field-id based column
resolution)
Testing
* added e2e test 'iceberg-orc-field-id.test' by copying the contents of
nested-types-scanner-basic,
nested-types-scanner-array-materialization,
nested-types-scanner-position,
nested-types-scanner-maps,
and executing the queries on an Iceberg table with ORC data files
Change-Id: Ia2b1abcc25ad2268aa96dff032328e8951dbfb9d
---
M be/src/exec/orc-metadata-utils.cc
M be/src/exec/orc-metadata-utils.h
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/util/debug-util.cc
M be/src/util/debug-util.h
M bin/impala-config.sh
M common/thrift/Query.thrift
M testdata/data/README
A
testdata/data/iceberg_test/hadoop_catalog/ice/complextypestbl_iceberg_orc/data/00000-0-boroknagyz_20210331133358_b718b2ff-9f49-4056-a5ed-0d37ec144fff-job_16171873329050_0002-00001.orc
A
testdata/data/iceberg_test/hadoop_catalog/ice/complextypestbl_iceberg_orc/data/00001-0-boroknagyz_20210331133358_b718b2ff-9f49-4056-a5ed-0d37ec144fff-job_16171873329050_0002-00001.orc
A
testdata/data/iceberg_test/hadoop_catalog/ice/complextypestbl_iceberg_orc/metadata/46b4a907-2ff3-4799-ba4a-074d04734265-m0.avro
A
testdata/data/iceberg_test/hadoop_catalog/ice/complextypestbl_iceberg_orc/metadata/snap-8747481058330439933-1-46b4a907-2ff3-4799-ba4a-074d04734265.avro
A
testdata/data/iceberg_test/hadoop_catalog/ice/complextypestbl_iceberg_orc/metadata/v1.metadata.json
A
testdata/data/iceberg_test/hadoop_catalog/ice/complextypestbl_iceberg_orc/metadata/v2.metadata.json
A
testdata/data/iceberg_test/hadoop_catalog/ice/complextypestbl_iceberg_orc/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-orc-field-id.test
M tests/query_test/test_iceberg.py
22 files changed, 2,888 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17398/2
--
To view, visit http://gerrit.cloudera.org:8080/17398
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia2b1abcc25ad2268aa96dff032328e8951dbfb9d
Gerrit-Change-Number: 17398
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: wangsheng <sky...@163.com>
