HyukjinKwon commented on code in PR #37531:
URL: https://github.com/apache/spark/pull/37531#discussion_r949739935
##########
core/src/main/scala/org/apache/spark/TaskContextImpl.scala:
##########
@@ -191,20 +191,69 @@ private[spark] class TaskContextImpl(
}
}
- val errorMsgs = new ArrayBuffer[String](2)
+ val listenerExceptions = new ArrayBuffer[Throwable](2)
var listenerOption: Option[T] = None
while ({listenerOption = getNextListenerOrDeregisterThread();
listenerOption.nonEmpty}) {
val listener = listenerOption.get
try {
callback(listener)
} catch {
case e: Throwable =>
- errorMsgs += e.getMessage
+ // A listener failed. Temporarily clear the listenerInvocationThread
and markTaskFailed.
+ //
+ // One of the following cases applies (#3 being the interesting one):
+ //
+ // 1. [[Task.doRunTask]] is currently calling [[markTaskFailed]]
because the task body
Review Comment:
No biggie but we could use backticks instead of `[[..]]` that is for
scaladoc (whereas here's a comment)
##########
core/src/main/scala/org/apache/spark/TaskContext.scala:
##########
@@ -133,21 +134,53 @@ abstract class TaskContext extends Serializable {
}
/**
- * Adds a listener to be executed on task failure. Adding a listener to an
already failed task
- * will result in that listener being called immediately.
+ * Adds a listener to be executed on task failure (which includes completion
listener failure, if
+ * the task body did not already fail). Adding a listener to an already
failed task will result in
+ * that listener being called immediately.
+ *
+ * Note: Prior to Spark 3.4.0, failure listeners were only invoked if the
main task body failed.
Review Comment:
nit: we have `@note` for a note but no biggie
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
