Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18605
Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column ...................................................................... IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column Identity-partitioned columns are not necessarily stored in the data files. E.g. when we migrate a legacy partitioned table to Iceberg without rewriting the data files, the partition columns won't be present in the files. The Parquet scanner does a few optimizations to eliminate row groups, i.e. filtering based on stats, bloom filters, etc. When a column is not present in the data file that has some predicate on, then it is assumed that the whole row group doesn't pass the filtering criteria. But for Iceberg some files might contain partition columns, while other files doesn't, so we need to prepare the scanners to handle such cases. The ORC scanner doesn't have that many optimizations so it didn't ran into this issue. Testing: * e2e tests Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4 --- M be/src/exec/file-metadata-utils.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test 3 files changed, 142 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/18605/1 -- To view, visit http://gerrit.cloudera.org:8080/18605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4 Gerrit-Change-Number: 18605 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
