(spark) branch master updated: [SPARK-45013][TEST] Flaky Test with NPE: track allocated resources by taskId

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 1370e9de0e4 [SPARK-45013][TEST] Flaky Test with NPE: track allocated 
resources by taskId
1370e9de0e4 is described below

commit 1370e9de0e49012f2886aedb30da5117e37f2ef1
Author: Kent Yao <y...@apache.org>
AuthorDate: Mon Nov 6 22:53:12 2023 -0800

    [SPARK-45013][TEST] Flaky Test with NPE: track allocated resources by taskId
    
    ### What changes were proposed in this pull request?
    
    This PR ensures the runningTasks to be updated before subsequent tasks 
causing NPE
    
    ### Why are the changes needed?
    
    fix flakey tests
    
    ### Does this PR introduce _any_ user-facing change?
    
    no
    ### How was this patch tested?
    
    It shall fix
    ```
    - track allocated resources by taskId *** FAILED *** (36 milliseconds)
    [info]   java.lang.NullPointerException: Cannot invoke 
"org.apache.spark.executor.Executor$TaskRunner.taskDescription()" because the 
return value of "java.util.concurrent.ConcurrentHashMap.get(Object)" is null
    [info]   at 
org.apache.spark.executor.CoarseGrainedExecutorBackend.statusUpdate(CoarseGrainedExecutorBackend.scala:275)
    [info]   at 
org.apache.spark.executor.CoarseGrainedExecutorBackendSuite.$anonfun$new$22(CoarseGrainedExecutorBackendSuite.scala:351)
    [info]
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    no
    
    Closes #43693 from yaooqinn/SPARK-45013.
    
    Authored-by: Kent Yao <y...@apache.org>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala    | 1 +
 1 file changed, 1 insertion(+)

diff --git 
a/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
 
b/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
index 28af0656869..35fe0b0d1c9 100644
--- 
a/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
+++ 
b/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
@@ -343,6 +343,7 @@ class CoarseGrainedExecutorBackendSuite extends 
SparkFunSuite
       backend.self.send(LaunchTask(new 
SerializableBuffer(serializedTaskDescription)))
       eventually(timeout(10.seconds)) {
         assert(backend.taskResources.size == 1)
+        assert(runningTasks.size == 1)
         val resources = backend.taskResources.get(taskId)
         assert(resources(GPU).addresses sameElements Array("0", "1"))
       }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org