nevzheng commented on issue #3675:
URL: https://github.com/apache/iceberg-rust/issues/3675#issuecomment-6832303217

   Thanks for writing this up, @Perseus, and for the draft in #3674.
   
   The use case makes sense to me: engines learn things during execution (e.g. 
DataFusion dynamic filters from a join build side) that could prune whole files 
by partition, and today there's no supported way to do that without 
reimplementing internals, as the DataFusion Distributed draft shows.
   
   **High level:** From my perspective, this feels like a new public API for 
iceberg-rust, so it seems worth taking extra care in how it's designed and 
maintained. For that reason I lean toward Approach 2: a small, deliberate 
surface rather than making `ExpressionEvaluator` / `InclusiveProjection` 
public. I'm not sure if those were intended as public contracts, and exposing 
them would push partition-schema construction and spec evolution onto callers 
and lock in internals that may need to change.
   
   **A possible way to approach it:**
   1. **Start with the minimal subset this use case needs:** given a predicate, 
decide whether a `FileScanTask` can be skipped based on its partition values. 
Define that well and keep the rest internal.
   2. **Align on scope before expanding.** In the community sync, Matt raised 
questions about which kinds of operations/predicates this should support. A 
short scope doc (what's in v1, what's deliberately out) might help settle that 
up front rather than in PR review, though I'll defer to the maintainers on 
whether that's needed.
   3. **Treat extensions as follow-ups:** e.g. more predicate types, row-group 
or metrics-based pruning, or a built-in runtime-filter hook in the scan itself. 
As use cases emerge etc.
   
   **Things a scope doc could cover:**
   - **Use cases / CUJs:** the concrete callers this needs to serve (e.g. 
DataFusion dynamic filters, DataFusion Distributed, eager-planning engines 
similar to Spark's use of Java's `Evaluator`/`Projection`), and which are in 
scope for v1 vs. later.
   - **API details:**
     - *Semantics/naming:* inclusive projection answers "might match", so 
`false` is safe to skip on and `true` means "maybe". Something like 
`might_match` could make that conservative contract more explicit.
     - *Construction cost:* projection is per partition spec, not per task. 
Building the filter once per scan (predicate + schema + case sensitivity) and 
caching projections by `spec_id` internally could help, and would also hide 
spec evolution from callers.
     - *Signatures:* taking things callers already have (unbound `Predicate`, 
`FileScanTask`) would keep internal types out of the public surface.
     - *Edge cases:* predicates on non-partition columns, non-projectable 
transforms, unknown specs. My instinct is that these should degrade to "don't 
skip" rather than erroring.
   - **Test plan / validation:** the key invariant is no false negatives (never 
skip a file that could contain matching rows). Coverage across transforms 
(identity, bucket, truncate, year/month/day/hour, void), tables with multiple 
partition specs, null partition values, and case sensitivity seems important. 
Possibly also parity checks against Java's `Evaluator` behavior, plus an 
end-to-end test showing files actually get skipped in an engine integration.
   - **Maintainers / ownership:** who reviews and owns this surface going 
forward, since it'll be a contract downstream engines depend on.
   - **Evolution:** how it can grow without breaking callers (e.g. 
`#[non_exhaustive]`, builder-style construction, possibly marking it 
experimental at first), and how it relates to a future built-in runtime-filter 
hook. Ideally iceberg-rust's own planning uses the same code path, so the API 
stays well tested.
   
   Happy to help review as this evolves. Thanks for driving it!
   
   cc: @mbutrovich 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to