tustvold commented on PR #6676: URL: https://github.com/apache/arrow-rs/pull/6676#issuecomment-2551016464
FWIW I did have some ideas about exposing lower level APIs in https://github.com/apache/arrow-rs/issues/5522 that would allow finer grained control of decode but I've not had time, and am unlikely to have time, to write up a concrete proposal. I agree that we should provide a way to concurrently fetch IO and do parallel decode, it is unclear that this can be made to work in the RecordBatchStream interface, or if it would need a new API as proposed in #5522 _On a historical note, RecordBatchStream was itself a compromise, I originally wanted a push-based decoder https://github.com/apache/arrow-rs/issues/1605 but in the interests of time I went with the simpler, but more constraining option._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
