nevzheng commented on issue #3675:
URL: https://github.com/apache/iceberg-rust/issues/3675#issuecomment-6832303217
Thanks for writing this up, @Perseus, and for the draft in #3674.
The use case makes sense to me: engines learn things during execution (e.g.
DataFusion dynamic filters from a join build side) that could prune whole files
by partition, and today there's no supported way to do that without
reimplementing internals, as the DataFusion Distributed draft shows.
**High level:** From my perspective, this feels like a new public API for
iceberg-rust, so it seems worth taking extra care in how it's designed and
maintained. For that reason I lean toward Approach 2: a small, deliberate
surface rather than making `ExpressionEvaluator` / `InclusiveProjection`
public. I'm not sure if those were intended as public contracts, and exposing
them would push partition-schema construction and spec evolution onto callers
and lock in internals that may need to change.
**A possible way to approach it:**
1. **Start with the minimal subset this use case needs:** given a predicate,
decide whether a `FileScanTask` can be skipped based on its partition values.
Define that well and keep the rest internal.
2. **Align on scope before expanding.** In the community sync, Matt raised
questions about which kinds of operations/predicates this should support. A
short scope doc (what's in v1, what's deliberately out) might help settle that
up front rather than in PR review, though I'll defer to the maintainers on
whether that's needed.
3. **Treat extensions as follow-ups:** e.g. more predicate types, row-group
or metrics-based pruning, or a built-in runtime-filter hook in the scan itself.
As use cases emerge etc.
**Things a scope doc could cover:**
- **Use cases / CUJs:** the concrete callers this needs to serve (e.g.
DataFusion dynamic filters, DataFusion Distributed, eager-planning engines
similar to Spark's use of Java's `Evaluator`/`Projection`), and which are in
scope for v1 vs. later.
- **API details:**
- *Semantics/naming:* inclusive projection answers "might match", so
`false` is safe to skip on and `true` means "maybe". Something like
`might_match` could make that conservative contract more explicit.
- *Construction cost:* projection is per partition spec, not per task.
Building the filter once per scan (predicate + schema + case sensitivity) and
caching projections by `spec_id` internally could help, and would also hide
spec evolution from callers.
- *Signatures:* taking things callers already have (unbound `Predicate`,
`FileScanTask`) would keep internal types out of the public surface.
- *Edge cases:* predicates on non-partition columns, non-projectable
transforms, unknown specs. My instinct is that these should degrade to "don't
skip" rather than erroring.
- **Test plan / validation:** the key invariant is no false negatives (never
skip a file that could contain matching rows). Coverage across transforms
(identity, bucket, truncate, year/month/day/hour, void), tables with multiple
partition specs, null partition values, and case sensitivity seems important.
Possibly also parity checks against Java's `Evaluator` behavior, plus an
end-to-end test showing files actually get skipped in an engine integration.
- **Maintainers / ownership:** who reviews and owns this surface going
forward, since it'll be a contract downstream engines depend on.
- **Evolution:** how it can grow without breaking callers (e.g.
`#[non_exhaustive]`, builder-style construction, possibly marking it
experimental at first), and how it relates to a future built-in runtime-filter
hook. Ideally iceberg-rust's own planning uses the same code path, so the API
stays well tested.
Happy to help review as this evolves. Thanks for driving it!
cc: @mbutrovich
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]