(spark) branch master updated: [SPARK-46344][CORE] Warn properly when a driver exists successfully but master is disconnected

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 166df8b1df59 [SPARK-46344][CORE] Warn properly when a driver exists 
successfully but master is disconnected
166df8b1df59 is described below

commit 166df8b1df5965fe3f15fecd5574545746b0b18f
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Sat Dec 9 15:33:48 2023 -0800

    [SPARK-46344][CORE] Warn properly when a driver exists successfully but 
master is disconnected
    
    ### What changes were proposed in this pull request?
    
    This PR aims to warn properly when a driver exists successfully but master 
is disconnected.
    
    ### Why are the changes needed?
    
    In this case, `Master` considers them `Error` eventually.
    
    ![Screenshot 2023-12-09 at 3 05 27 
PM](https://github.com/apache/spark/assets/9700541/1323819b-4a0c-466d-afaa-845f507a905e)
    
    **Worker Log**
    ```
    23/12/09 15:13:21 INFO Worker: Driver driver-20231209151301-0003 exited 
successfully
    === Master is disconnected here ===
    23/12/09 15:13:53 WARN Worker: Driver driver-20231209151332-0004 exited 
successfully while master is disconnected.
    === A new master starts and is connected here ===
    23/12/09 15:17:10 INFO Worker: Driver driver-20231209151707-0000 exited 
successfully
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Pass the CIs.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #44278 from dongjoon-hyun/SPARK-46344.
    
    Authored-by: Dongjoon Hyun <dh...@apple.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala 
b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
index 1422a1484f8d..785129e1d818 100755
--- a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
@@ -878,7 +878,12 @@ private[deploy] class Worker(
       case DriverState.FAILED =>
         logWarning(s"Driver $driverId exited with failure")
       case DriverState.FINISHED =>
-        logInfo(s"Driver $driverId exited successfully")
+        registrationRetryTimer match {
+          case Some(_) =>
+            logWarning(s"Driver $driverId exited successfully while master is 
disconnected.")
+          case _ =>
+            logInfo(s"Driver $driverId exited successfully")
+        }
       case DriverState.KILLED =>
         logInfo(s"Driver $driverId was killed by user")
       case _ =>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org