tustvold commented on issue #1955: URL: https://github.com/apache/arrow-rs/issues/1955#issuecomment-1168467248
> Is there any need to support in ParquetRecordBatchReader, or They reuse a lot of logic between them (support one is like almost support both). They reuse a lot of logic, however, the logic that differs concerns the IO for fetching pages. So support for this would need to be explicitly added. > could you show me the code example of IOX integrate with arrow-rs Currently IOx fetches the entire file to memory and does not perform IO to object storage directly. This was partly driven by the limited support for more sophisticated predicate pushdown, and the fact IO was not a dominating factor for our query workloads. That being said, https://github.com/apache/arrow-datafusion/pull/2677 switches DataFusion to using the async interface directly, and https://github.com/apache/arrow-datafusion/issues/2504 has more about how I envisage this fitting with the rayon-based scheduler longer-term. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
