This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new e4e41b2bf6fc [SPARK-45013][CORE][TEST][3.5] Flaky Test with NPE: track
allocated resources by taskId
e4e41b2bf6fc is described below
commit e4e41b2bf6fc7afecd86d0e4da816487d3c03d1b
Author: Kent Yao <[email protected]>
AuthorDate: Tue Jan 21 12:51:45 2025 -0800
[SPARK-45013][CORE][TEST][3.5] Flaky Test with NPE: track allocated
resources by taskId
### What changes were proposed in this pull request?
This PR ensures the runningTasks to be updated before subsequent tasks
causing NPE
### Why are the changes needed?
fix flakey tests
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
It shall fix
```
- track allocated resources by taskId *** FAILED *** (36 milliseconds)
[info] java.lang.NullPointerException: Cannot invoke
"org.apache.spark.executor.Executor$TaskRunner.taskDescription()" because the
return value of "java.util.concurrent.ConcurrentHashMap.get(Object)" is null
[info] at
org.apache.spark.executor.CoarseGrainedExecutorBackend.statusUpdate(CoarseGrainedExecutorBackend.scala:275)
[info] at
org.apache.spark.executor.CoarseGrainedExecutorBackendSuite.$anonfun$new$22(CoarseGrainedExecutorBackendSuite.scala:351)
[info]
```
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #49589 from LuciferYang/SPARK-45013-3.5.
Authored-by: Kent Yao <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala | 1 +
1 file changed, 1 insertion(+)
diff --git
a/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
b/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
index 909d60544257..0b931840dd2a 100644
---
a/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
+++
b/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
@@ -339,6 +339,7 @@ class CoarseGrainedExecutorBackendSuite extends
SparkFunSuite
backend.self.send(LaunchTask(new
SerializableBuffer(serializedTaskDescription)))
eventually(timeout(10.seconds)) {
assert(backend.taskResources.size == 1)
+ assert(runningTasks.size == 1)
val resources = backend.taskResources.get(taskId)
assert(resources(GPU).addresses sameElements Array("0", "1"))
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
