Hello Daniel Becker, Gabor Kaszab, Mihaly Szjatinya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21761
to look at the new patch set (#7).
Change subject: IMPALA-13364: Schema resolution doesn't work for migrated
partitioned Iceberg tables that have complex types
......................................................................
IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg
tables that have complex types
Schema resolution doesn't work correctly for migrated partitioned
Iceberg tables that have complex types. When we face a Parquet/ORC file
in an Iceberg table that doesn't have field IDs in the file metadata, we
assume that it is an old data file before migration, and the schema is
the very first one, hence we can mimic Iceberg's field ID generation to
assign field IDs to the file schema elements.
This process didn't take the partition columns into account. Partition
columns are not part of the data file but they still get field IDs. This
only matters when there are complex types in the table, as partition
columns are always the last columns in legacy Hive tables, and field IDs
are assigned via a "BFS-like" traversal. I.e. if there are only primitive
types in the table we don't have any problems, but the children of
complex types columns are assigned incorrectly.
This patch fixes field ID generation by taking the number of partitions
into account. If none of the partition columns are included in the data
file (common case) we adjust the file-level field IDs accordingly. It is
also OK to have all the partition columns in the data files (it is not
common, but we've seen such data files). We raise an error in other
cases (some partition columns are in the data file, while others
aren't).
Testing:
* e2e tests added
* added negative tests
Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/orc/orc-metadata-utils.cc
M be/src/exec/orc/orc-metadata-utils.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
A testdata/migrated_iceberg/README
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.orc
A
testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.parquet
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-table-field-id-resolution-complex.test
M tests/query_test/test_iceberg.py
13 files changed, 530 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/21761/7
--
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Daniel Becker <[email protected]>
Gerrit-Reviewer: Gabor Kaszab <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Mihaly Szjatinya <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
