LuciferYang opened a new issue, #13544: URL: https://github.com/apache/gluten/issues/13544
`VeloxRssSortShuffleWriter` keeps the string buffers it has already counted in `stringBuffers_`, so that a buffer shared by several vectors is counted once in `currentInputColumnBytes_`. `reclaimFixedSize` evicts the buffered batches and resets the byte counter, but it does not clear `stringBuffers_`. After a reclaim, a string buffer that is still alive, or a new buffer allocated at a freed address, is treated as already counted. The next batches then under-report their size, and the writer evicts later than its memory threshold. `resetBatches`, used after a normal eviction, clears all three. This writer is used with Celeborn when `spark.celeborn.client.spark.shuffle.writer=sort`. This issue was written with the assistance of AI (Claude Opus). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
