comphead commented on code in PR #7638:
URL: https://github.com/apache/datafusion-comet/pull/7638#discussion_r4608077360


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala:
##########
@@ -485,7 +491,13 @@ class CometNativeShuffleWriter[K, V](
 
   override def stop(success: Boolean): Option[MapStatus] = {
     remoteDestination match {
-      case None => if (success) Some(mapStatus) else None
+      case None =>
+        if (success) {
+          Some(mapStatus)
+        } else {
+          discardLocalTemp(None)

Review Comment:
   This `discardLocalTemp(None)` repeats the cleanup in the `catch` of `write` 
(line 114). Spark's `ShuffleWriteProcessor` calls `stop(false)` only after 
`write` threw an `Exception`, and the `catch` here already handles every 
`Throwable`. `localTempData` is only set inside `writeInternal`, so I could not 
find a path where `stop(false)` sees a temp file that `write` did not already 
delete. `CometBypassMergeSortShuffleWriter` follows the same rule and cleans up 
in `write` only. Dropping this branch and the `stop(false)` test would leave 
one cleanup path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to