westonpace commented on PR #41801: URL: https://github.com/apache/arrow/pull/41801#issuecomment-2128266456
+1 for the PR I think 4MiB is the right default. > I'm not very familiar with the C++ IO abstractions, but within arrow-rs, in particular object_store, we support native streaming reads I think this is something of a moot point since we are generally talking about `get_range` and not `get`. The C++ filesystem library has no concept of `get`. I believe, even in Rust, that users still need to make their own choice about what size ranges they are using when they are doing byte ranges. I don't think there is any Rust equivalent of `CompressedInputStream`. This is primarily used in arrow-cpp for CSV reading (and maybe compressed IPC buffers?) I believe Rust's IPC decompression is "whole buffer" and not streaming. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
