Hello Balazs Hevele, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24553
to look at the new patch set (#6).
Change subject: POC: IMPALA-15163 Add geometry bounding box predicate pushdown
for Parquet
......................................................................
POC: IMPALA-15163 Add geometry bounding box predicate pushdown for Parquet
Implement Parquet row group skipping based on GEOMETRY column
bounding box statistics. Iceberg does not support predicate
pushdown for geometry types yet (see
https://github.com/apache/iceberg/pull/14101), so this operates
at the Parquet level using column chunk min/max stats directly.
Note that pruning during planning would be still possible in Impala -
bounds are available in Iceberg's DataFile metadata returned by
planFiles() - but this is not implemented in this commit.
Writer side: the Parquet table writer computes per-column bbox
(xmin||ymin as min_value, xmax||ymax as max_value, 16 bytes each
as LE doubles) by tracking the envelope of all geometries written
to each column chunk. The statistics propagate to Iceberg manifests
via the normal commit path.
Reader side: HdfsScanNode.tryComputeGeoBBoxPredicate() recognizes
st_envintersects(CONST, geom_col) conjuncts, computes the constant
geometry's envelope via ESRI, and emits TGeoBBoxPredicate to the
backend. EvaluateGeoBBoxPredicates() in the Parquet scanner checks
bbox overlap against row group column stats (not Iceberg metadata)
and increments NumStatsFilteredRowGroups on miss.
Also fixes Iceberg metadata table queries (e.g. data_files) for
tables with GEOMETRY columns: excludes 'readable_metrics' from the
scan projection because Iceberg 1.10.2's Conversions.fromByteBuffer
does not handle GEOMETRY (throws UnsupportedOperationException).
The column returns NULL until a future Iceberg release adds support.
Testing:
- create table with Impala and verify that bbox is written and used
- check relations that do pushdown (st_envintersects, etc..) and
st_disjoint that must not do pushdown
- use metadata table queries to verify bbox stats are written to
Iceberg manifests
- no interop tests yet, would need to find an engine that writes
statistics
Change-Id: I7265a00caba698a122bc15ecb433c507f36d166f
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.h
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/rewrite/PointEnvIntersectsRule.java
M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java
M
java/hive-geospatial-functions/src/main/java/org/apache/impala/hive/geospatial/esri/GeometryUtils.java
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry-bbox.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry.test
M tests/custom_cluster/test_geospatial_library.py
12 files changed, 618 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/24553/6
--
To view, visit http://gerrit.cloudera.org:8080/24553
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7265a00caba698a122bc15ecb433c507f36d166f
Gerrit-Change-Number: 24553
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Balazs Hevele <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
