yew1eb opened a new pull request, #3815: URL: https://github.com/apache/celeborn/pull/3815
### What changes were proposed in this pull request? Add the missing `endMapIndex == Integer.MAX_VALUE` clamp in `ShuffleBlockInfoUtils.sliceSortedBufferByMapRange`, mirroring the existing guard in the sibling method `getChunkOffsetsFromShuffleBlockInfos`. Also add a unit test covering the full re-read (`Integer.MAX_VALUE`) case. ### Why are the changes needed? When a memory-stored shuffle partition is read once with a map range (e.g. AQE skew join) and then re-read in full (`endIndex = Int.MaxValue`, e.g. Spark ReusedExchange), `FetchHandler.handleReduceOpenStreamInternal` passes the raw `Int.MaxValue` into `getSortedFileInfo` because `FileInfo.addStream` returns false for the already-sorted file. The memory branch then loops `startMapIndex..Integer.MAX_VALUE` in `sliceSortedBufferByMapRange` — billions of `TreeMap.get` calls on the Netty fetch event loop, stalling all fetch/openStream traffic on that event loop for minutes to hours. The disk path is not affected (it goes through the already-clamped `getChunkOffsetsFromShuffleBlockInfos`). ### Does this PR resolve a correctness bug? - [x] Yes ### Does this PR introduce _any_ user-facing change? - [ ] Yes ### How was this patch tested? - Added `testSliceSortedBufferByMapRangeWithMaxEndIndex` to `ShuffleBlockInfoUtilsTest`, asserting a full re-read with `Integer.MAX_VALUE` produces exactly the same result as an explicit full-range read. - `ShuffleBlockInfoUtilsTest`: all 6 tests pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
