gaoyajun02 commented on code in PR #56559:
URL: https://github.com/apache/spark/pull/56559#discussion_r3465180173
##########
core/src/main/scala/org/apache/spark/TaskContext.scala:
##########
@@ -195,6 +195,24 @@ abstract class TaskContext extends Serializable {
})
}
+ /**
+ * Adds a listener to be invoked after the task's status update has been
sent to the driver.
+ * This is useful for operations that should only begin after the driver has
been notified
+ * of the task's result. For example, push-based shuffle block push can use
this to
+ * ensure the driver processes the task result before any push data reaches
the merger,
+ * avoiding stale data being merged without detection.
+ *
+ * The callback runs on the same executor thread that sends the status
update.
+ */
+ @Experimental
+ def addPostStatusUpdateListener(listener: PostStatusUpdateListener):
TaskContext
Review Comment:
Agreed — changed `addPostStatusUpdateListener` from `public @Experimental`
to `private[spark]` to match the visibility of `PostStatusUpdateListener`. The
mechanism is entirely internal to push-based shuffle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
