Dandandan commented on PR #20481:
URL: https://github.com/apache/datafusion/pull/20481#issuecomment-3980294024
This suggestion from Claude seems like a logical next step to split more
work to morsels instead of during `open`:
```
Phase 1 — Metadata fetch (one partition, fast):
- Read footer once
- Run statistics-based row group pruning (in-memory, no I/O)
- Distribute row groups as "pre-pruned morsels" into the queue
Phase 2 — Per-row-group I/O (all partitions, concurrent):
- Each partition pulls a row group morsel from the queue
- Loads bloom filter for its row group
- Loads page index for its row group
- Reads the data
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
