uros-b commented on code in PR #56628:
URL: https://github.com/apache/spark/pull/56628#discussion_r3466165030
##########
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala:
##########
@@ -2921,6 +2921,82 @@ class TaskSetManagerSuite
s"\nCaptured logs:\n${logs.mkString("\n")}")
}
+ // SPARK-57465: Demonstrates that RejectedExecutionException (wrapped as
ExceptionFailure)
+ // counts toward task failure budget, consuming retries for a non-task-fault.
+ test("SPARK-57465: RejectedExecutionException counts toward task failures
(bug repro)") {
+ sc = new SparkContext("local", "test")
+ sched = new FakeTaskScheduler(sc, ("exec1", "host1"))
+ val taskSet = FakeTask.createTaskSet(1)
+ val clock = new ManualClock(1) // start at 1 so markFinished(time > 0)
passes
+ val manager = new TaskSetManager(sched, taskSet, MAX_TASK_FAILURES, clock
= clock)
+
+ // Simulate the failure that Executor.launchTask sends when
threadPool.execute throws
+ // RejectedExecutionException: it wraps as ExceptionFailure (which has
+ // countTowardsTaskFailures = true by default).
+ val rejectedExc = new java.util.concurrent.RejectedExecutionException(
+ "Task org.apache.spark.executor.Executor$TaskRunner rejected from " +
+ "java.util.concurrent.ThreadPoolExecutor [Shutting down]")
+ val failureReason = new ExceptionFailure(
+ rejectedExc.getClass.getName,
+ rejectedExc.getMessage,
+ rejectedExc.getStackTrace,
+ org.apache.spark.util.Utils.exceptionString(rejectedExc),
+ None)
+
Review Comment:
This test is largely tautological: it hand-builds an ExceptionFailure and
asserts countTowardsTaskFailures === true, then drives TSM. It re-verifies
pre-existing TSM counting given a manually-constructed reason, not the PR's
Executor change. Please consider dropping it and folding the contrast into the
positive test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
