andygrove opened a new issue, #7488: URL: https://github.com/apache/datafusion-comet/issues/7488
## Describe the bug `CometShuffleManager` handles shuffles that Comet does not replace differently from Spark's `SortShuffleManager`. These shuffles are RDD shuffles and Spark exchanges Comet leaves in place. They are not new, but they reach more applications once the plugin sets `CometShuffleManager` by default on Spark 4.0 and later (#7336). Found in review of that PR. 1. **Extra row copy in Spark exchanges.** `ShuffleExchangeExec.needToCopyObjectsBeforeShuffle` returns true for any manager that is not a `SortShuffleManager`, so Spark's own exchanges copy every row before the shuffle. 2. **`sortShuffleManager.getWriter` entries are never released.** For a Spark shuffle dependency, `getWriter` delegates to the inner `sortShuffleManager`, which records each map task in its own `taskIdMapsForShuffle`. `unregisterShuffle` clears only `CometShuffleManager`'s map and never calls `sortShuffleManager.unregisterShuffle`, so those entries stay on the executor heap until the application ends. 3. **Blocks migrated in during decommissioning are never deleted.** The `IndexShuffleBlockResolver` is built with a fresh `ConcurrentHashMap` instead of `taskIdMapsForShuffle`. The resolver records blocks migrated in during decommissioning in that map, so `unregisterShuffle` never deletes them. All three are in `spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleManager.scala`. ## Steps to reproduce Read the code paths above. No reproduction has been run for any of them. ## Expected behavior A shuffle that Comet does not replace behaves as it does under `SortShuffleManager`: no extra row copy, and no state left behind after `unregisterShuffle`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
