Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21761 )
Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types ...................................................................... IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types Schema resolution doesn't work correctly for migrated partitioned Iceberg tables that have complex types. When we face a Parquet/ORC file in an Iceberg table that doesn't have field IDs in the file metadata, we assume that it is an old data file before migration, and the schema is the very first one, hence we can mimic Iceberg's field ID generation to assign field IDs to the file schema elements. This process didn't take the partition columns into account. Partition columns are not part of the data file but they still get field IDs. This only matters when there are complex types in the table, as partition columns are always the last columns in legacy Hive tables, and field IDs are assigned via a "BFS-like" traversal. I.e. if there are only primitive types in the table we don't have any problems, but the children of complex types columns are assigned incorrectly. This patch fixes field ID generation by taking the number of partitions into account. If none of the partition columns are included in the data file (common case) we adjust the file-level field IDs accordingly. It is also OK to have all the partition columns in the data files (it is not common, but we've seen such data files). We raise an error in other cases (some partition columns are in the data file, while others aren't). Testing: * e2e tests added * added negative tests Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b Reviewed-on: http://gerrit.cloudera.org:8080/21761 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/file-metadata-utils.cc M be/src/exec/file-metadata-utils.h M be/src/exec/orc/hdfs-orc-scanner.cc M be/src/exec/orc/orc-metadata-utils.cc M be/src/exec/orc/orc-metadata-utils.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h A testdata/migrated_iceberg/README A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.orc A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.parquet A testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-table-field-id-resolution-complex.test M tests/query_test/test_iceberg.py 13 files changed, 530 insertions(+), 15 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b Gerrit-Change-Number: 21761 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Gabor Kaszab <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Mihaly Szjatinya <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
