alamb commented on issue #24393: URL: https://github.com/apache/datafusion/issues/24393#issuecomment-5302232683
Some related issues for context: DataFusion (remaining CPU-side work): - https://github.com/apache/datafusion/issues/20325 arrow-rs (where the progressive fetch / I/O mechanism lives): - https://github.com/apache/arrow-rs/issues/8668 -- likely the closest existing upstream hook for fetching all pages up front with the push decoder - https://github.com/apache/arrow-rs/issues/7983 -- the push decoder work that decouples I/O from decode, making an I/O policy split possible arrow-rs work showing predicate evaluation is now computationally the same or faster: - https://github.com/apache/arrow-rs/issues/7362 - https://github.com/apache/arrow-rs/issues/8203 Remaining upstream CPU-side work: - https://github.com/apache/arrow-rs/issues/7456 - https://github.com/apache/arrow-rs/issues/9589 A prior concrete example of the I/O pattern problem described here: - https://github.com/apache/arrow-rs/issues/8565 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
