alamb commented on issue #10073: URL: https://github.com/apache/arrow-rs/issues/10073#issuecomment-4653116396
> 15GB to 30MB is crazy! do you have any performance numbers? I wonder if it's faster (because memory) or slower (because more memory fragmentation, etc). My very unscientific measurements on https://github.com/alamb/wide-parquet show it is slower (12%) ``` Total elapsed time : 1.172 s ``` vs ``` Total elapsed time : 1.336 s ``` This is almost certainly due to the new small IOs that got inserted. I am sure with some more thoughtful IO optimizations (larger speculative reads and buffering) you could probably minimize almost all the overhead -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
