andygrove opened a new issue, #7070: URL: https://github.com/apache/datafusion-comet/issues/7070
### What is the problem the feature request solves? `NativeBatchDecoderIterator` reallocates its direct read buffer for every map-output block larger than 128 KiB. Each reallocation is a zeroing `allocateDirect` plus `Bits.reserveMemory`, which can force a `System.gc()` under direct-memory pressure. `CometBlockStoreShuffleReader.read()` creates one `NativeBatchDecoderIterator` per fetched block and closes it when the block is done. `close()` calls `resetDataBuf()`, which replaces the thread-local buffer with a fresh 128 KiB one whenever it has grown past `INITIAL_BUFFER_SIZE`. The next block over 128 KiB grows it again. So a shuffle whose compressed blocks exceed 128 KiB (#5678 measured 700 KiB to 1.4 MiB per block at the default 8192-row batch size) pays two `allocateDirect` calls per block. The sibling `CometShuffleBlockIterator.java` keeps its buffer across blocks and never resets. This path is used by the JVM-consumer read (decoded batches imported into Arrow Java vectors) and by every non-direct read, so it matters for stages that end in a Spark operator and for reads without AQE (#6807). ### Describe the potential solution Own the buffer per task instead of per iterator: a task-scoped growable direct buffer shared by the iterators that `CometBlockStoreShuffleReader.read()` creates, released in the task completion listener that `read()` already registers. Drop the thread-local and `resetDataBuf()`. Add a lifecycle check to `CometCelebornShuffleReaderSuite` (the decoder lifecycle checks live there) and an end-to-end shuffle-read measurement with blocks above 128 KiB. ### Additional context Finding R5 in #5905, originally item 2 of #5198. An earlier attempt (#5913) was closed unmerged, so there is no open PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
