Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24553
Change subject: POC: Add geometry bounding box predicate pushdown for Parquet ...................................................................... POC: Add geometry bounding box predicate pushdown for Parquet Implement Parquet row group skipping based on GEOMETRY column bounding box statistics. Iceberg does not support predicate pushdown for geometry types yet (see https://github.com/apache/iceberg/pull/14101), so this operates at the Parquet level using column chunk min/max stats directly. Note that pruning during planning would be still possible in Impala - bounds are available in Iceberg's DataFile metadata returned by planFiles() - but this is not implemented in this commit. Writer side: the Parquet table writer computes per-column bbox (xmin||ymin as min_value, xmax||ymax as max_value, 16 bytes each as LE doubles) by tracking the envelope of all geometries written to each column chunk. The statistics propagate to Iceberg manifests via the normal commit path. Reader side: HdfsScanNode.tryComputeGeoBBoxPredicate() recognizes st_envintersects(CONST, geom_col) conjuncts, computes the constant geometry's envelope via ESRI, and emits TGeoBBoxPredicate to the backend. EvaluateGeoBBoxPredicates() in the Parquet scanner checks bbox overlap against row group column stats (not Iceberg metadata) and increments NumStatsFilteredRowGroups on miss. Also fixes Iceberg metadata table queries (e.g. data_files) for tables with GEOMETRY columns: excludes 'readable_metrics' from the scan projection because Iceberg 1.10.2's Conversions.fromByteBuffer does not handle GEOMETRY (throws UnsupportedOperationException). The column returns NULL until a future Iceberg release adds support. Testing: - create table with Impala and verify that bbox is written and used - check relations that do pushdown (st_envintersects, etc..) and st_disjoint that must not do pushdown - use metadata table queries to verify bbox stats are written to Iceberg manifests - no interop tests yet, would need to find an engine that writes statistics Change-Id: I7265a00caba698a122bc15ecb433c507f36d166f --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/hdfs-parquet-table-writer.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/hive/geospatial/esri/GeometryUtils.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/rewrite/PointEnvIntersectsRule.java M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry-bbox.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry.test M tests/custom_cluster/test_geospatial_library.py 12 files changed, 618 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/24553/5 -- To view, visit http://gerrit.cloudera.org:8080/24553 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7265a00caba698a122bc15ecb433c507f36d166f Gerrit-Change-Number: 24553 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Balazs Hevele <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]>
