ozankabak commented on PR #4525: URL: https://github.com/apache/arrow-datafusion/pull/4525#issuecomment-1339579765
> What was the motivation for using ChunkedStore over just using the standard `LocalFileSystem`? This would return `GetResult::File` which would then synchronously read data from the file in batches automatically? Yes, I agree that this makes more sense in a real application. We are currently writing POCs/unit tests for candidate streaming execution approaches that operate on things like FIFOs, and we are using ChunkedStore because we saw it being used in tests and it is easy to control it to investigate things as you mention. When we first started, the context and intended usage of ChunkedStore was not entirely clear to us, and @metesynnada's first remark kind of tries to get clarification on that. Thanks for making it clear! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
