huan233usc opened a new pull request, #17175:
URL: https://github.com/apache/iceberg/pull/17175
> [!IMPORTANT]
> Draft / proof-of-concept. Standalone off `main`. This validates the
Expression-side
> integration for spatial file pruning; the API shape (predicate, operation
name,
> factory) is a sketch for discussion, not a final proposal. Not intended to
merge as-is.
## Why
The geometry/geography bounding-box metrics work makes a per-file spatial
bbox available
in `lower_bounds`/`upper_bounds`, and `GeospatialPredicateEvaluators` can
already
intersect two bounding boxes (including antimeridian wraparound for
geography). What is
missing is the connective tissue: nothing in the `Expression` API can
express a spatial
filter, and `InclusiveMetricsEvaluator` (the file-pruning engine used by
scan planning)
has no geo branch. This PoC adds that connective tissue on the core side and
proves a
spatial predicate prunes files end to end.
## What this adds (core / `api`)
- `Expression.Operation.ST_INTERSECTS`.
- `UnboundSpatialPredicate` / `BoundSpatialPredicate` — a predicate whose
constant is a
query `BoundingBox` (a spatial window) rather than a scalar `Literal`, so
it is modeled
as its own predicate kind instead of forcing a `BoundingBox` through
`Literals`.
- `Expressions.stIntersects(name, BoundingBox)` factory.
- `ExpressionVisitors`: `spatialPredicate(...)` visitor hooks plus dispatch
in `visit()`
and `visitEvaluator()`; `Binder` and `RewriteNot` handle the new predicate
so it binds
and traverses like any other.
- `InclusiveMetricsEvaluator`: reads the file's geometry/geography
`lower_bounds` /
`upper_bounds` (`Conversions.fromByteBuffer` → `GeospatialBound` →
`BoundingBox`) and
calls `GeospatialPredicateEvaluators.create(type).intersects(query,
fileBbox)`; a
disjoint file returns `ROWS_CANNOT_MATCH`.
## What it shows
`TestSpatialMetricsEvaluator` builds two files with disjoint geometry bounds
and binds a
real `ST_INTERSECTS` expression:
- a query window overlapping file A → `eval(A) == true` (scan), `eval(B) ==
false` (skip);
- a query window overlapping file B → the reverse.
This is the actual scan-planning path (`Expression` → `Binder` →
`InclusiveMetricsEvaluator.eval(file)`),
not a direct evaluator call.
## Still to do (follow-on, not in this commit yet)
- **Spark pushdown.** Spark 4.1 has no `st_intersects` (only WKB conversion
functions), so
Iceberg would register its own `iceberg.st_intersects` `ScalarFunction`,
and
`SparkV2Filters` would need a new branch to convert a top-level boolean
UDF predicate
into `Expressions.stIntersects(...)` (today it only converts UDFs as terms
inside
comparisons like `bucket(col) = 5`).
- Manifest-level pruning (`ManifestGroup`/partition bounds), predicate
serialization,
and the geography producer for the bbox metrics.
Opening as a draft to anchor the spatial-predicate API discussion and
confirm the pruning
approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
