steveloughran commented on PR #1139: URL: https://github.com/apache/parquet-mr/pull/1139#issuecomment-2021171417
> @steveloughran @mukund-thakur do you guys have any information on how much (if any) this impacts the peak memory utilization in the parquet file reader? The total memory allocated while reading a row group should remain the same, but if vector io allocates intermediate buffers (or even if the http request to S3, say, allocates additional memory) then we could see an increase in peak memory usage. @parthchandra for S3A reading into on-heap buffers (which is the only option this PR supports) we allocate in the SDK and just stream it in from the GET request. For reading into a merged range we discard the stuff in between. At the end of reading all the ranges then, they are all on heap and there shouldn't be extra overhead. During the read more connections are used in a shared thread pool; connection pool size shoulds to be increased as should the number of threads. We increase the defaults in 3.4.0; it's generally good to have big numbers. That is the biggest intermediate load penalty, as far as I can tell. For raw local: uses java nio API, so we hope they've got that right. And for the base implementation, it's just readFully() with offsets and range: nothing unusual. The off/heap story is a bit more complicated, which is partially why it is not supported here. For s3a data is streamed to heap in 2MB blocks and then copied off heap, so memory cost for no tangible gain. Java nio is better for local, but it becomes tricky to manage the lifespan of those buffers once range coalescing gets involved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
