adriangb opened a new issue, #16000:
URL: https://github.com/apache/datafusion/issues/16000
### Is your feature request related to a problem or challenge?
As we continue to make progress landing dynamic filters it opens up the
opportunity for new optimizations.
This one deals with late evaluation of file-level statistics.
In particular, we *may* have file level statics available at planning time
(see `datafusion.execution.collect_statistics` - our system does a similar
thing in a different way).
Before dynamic filters there was no point in re-evaluating these right
before scanning a file but now it's possible that e.g. a TopK operator passed
down a `ts > '2025-05-08T00:00:00Z'` filter -> we may be able to exclude the
entire file based on this filter + file level statistics -> we avoid reading
any Parquet metadata, etc.
In particular, change:
```rust
trait FileSource {
fn open(file_meta: FileMeta) ...
```
To:
```rust
trait FileSource {
fn open(file_meta: FileMeta, file: &PartitionedFile) ...
```
And call it as so from here:
https://github.com/pydantic/datafusion/blob/649851d59cdac80fcae51d66f82f1b47d2aaa3b4/datafusion/datasource/src/file_stream.rs#L129
Then we can implement
[PruningStatitics](https://github.com/pydantic/datafusion/blob/649851d59cdac80fcae51d66f82f1b47d2aaa3b4/datafusion/physical-optimizer/src/pruning.rs#L79)
for
[Statistics](https://github.com/pydantic/datafusion/blob/649851d59cdac80fcae51d66f82f1b47d2aaa3b4/datafusion/common/src/stats.rs#L270)
et. voilá!
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
