acvictor opened a new pull request, #13039: URL: https://github.com/apache/gluten/pull/13039
## What changes are proposed in this pull request? spark.shuffle.manager is process-wide, so ColumnarShuffleManager also serves the row-based exchanges Gluten does not offload. Spark decides whether each map task must defensively copy every row in ShuffleExchangeExec.needToCopyObjectsBeforeShuffle, which [gates on a single check, shuffleManager.isInstanceOf[SortShuffleManager]](https://github.com/apache/spark/blob/branch-4.1/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala#L297). Any other implementation falls through to a catch-all `true`. Because ColumnarShuffleManager extended ShuffleManager directly, that check was false and Spark selected the copying closure, running UnsafeRow.copy() once per row and allocating a tuple per row, instead of the branch that reuses a single MutablePair. The copy is pure overhead here rather than protection. It exists because ExternalSorter buffers deserialized rows, and the row-based branches in this class never reach that state: registerShuffle uses the same shouldBypassMergeSort and canUseSerializedShuffle predicates as SortShuffleManager, and getWriter builds the same BypassMergeSortShuffleWriter and UnsafeShuffleWriter. Same handle, same writer, same safety property; only the type check differed. Extending SortShuffleManager restores the zero-copy path. Shuffle output is unchanged. ## How was this patch tested? UT ## Was this patch authored or co-authored using generative AI tooling? Co-authored with GitHub Copilot CLI 1.0.84-4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
