unikdahal opened a new issue, #3336:
URL: https://github.com/apache/iceberg-rust/issues/3336
Raising this for tracking purpose rn.
I'm exploring extending the runtime-predicate direction discussed in #2376
so that a predicate can become more selective while a data file is already
being consumed.
#2376 discusses providing runtime predicates around `FileScanTask`
execution. The additional case I'm interested in is allowing the Arrow reader
to observe a newer predicate generation at row-group boundaries and prune row
groups that have not yet been read.
Roughly:
```text
RuntimePredicateProvider
|
|-- predicate snapshot
|-- generation
v
Arrow reader
|
v
refresh when generation changes
|
v
prune unread row groups / pages
```
The generation/version would avoid repeated work when the predicate has not
changed, while still allowing an execution-time producer to publish
progressively better bounds during an active scan.
I currently have a downstream prototype exercising this through DataFusion
Comet.
On a deliberately pruning-friendly Iceberg fixture with 4M rows, ~1.06 GB of
Parquet data, and 126 row groups, the prototype can prune 125 of 126 row groups
in selective cases, with roughly 96-99% lower reader-requested bytes depending
on the query.
If there is already work underway that can support this, I'd prefer to reuse
or extend it rather than introduce a parallel abstraction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
