Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19274 )
Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied ...................................................................... IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied If an Iceberg V2 table is partitioned, and contains delete files, then in a query that involves runtime filters on the partition columns return empty result set. E.g.: select count(*) from store_sales, date_dim where d_date_sk = ss_sold_date_sk and d_moy=2 and d_year=1998; In the above query store_sales is partitioned by ss_sold_date_sk which will be filtered by runtime filters created by the JOIN. If store_sales has delete files then the above query returns empty result set. The problem is that we are invoking PartitionPassesFilters() on these Iceberg tables. It is usually a no-op for Iceberg tables, as the template tuple is NULL. But when we have virtual columns a template tuple has been created in HdfsScanPlanNode::InitTemplateTuple. For Iceberg tables this tempalte tuple is incomplete, i.e. it doesn't have the partition values set. This means the filters evaluate to false and the files are getting filtered out, hence the query produces an empty result set. With this patch we don't invoke PartitionPassesFilters() on Iceberg tables, only the Iceberg-specific IcebergPartitionPassesFilters() gets invoked. Also added DCHECKs to ensure this. Testing: * e2e tests added Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3 Reviewed-on: http://gerrit.cloudera.org:8080/19274 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/orc/hdfs-orc-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test 5 files changed, 82 insertions(+), 18 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19274 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3 Gerrit-Change-Number: 19274 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Gergely Fürnstáhl <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tamas Mate <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
