[kyuubi] branch master updated: [KYUUBI #4936] [KYUUBI#4935][Improvement] More than target num of executors may survive after FinalStageResourceManager did kill

[chengpan]

This is an automated email from the ASF dual-hosted git repository.

chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git


The following commit(s) were added to refs/heads/master by this push:
     new 779e2937a [KYUUBI #4936] [KYUUBI#4935][Improvement] More than target 
num of executors may survive after FinalStageResourceManager did kill
779e2937a is described below

commit 779e2937a91326426c184809bc0672055ff41d44
Author: zhouyifan279 <zhouyifan...@gmail.com>
AuthorDate: Thu Jun 8 20:16:39 2023 +0800

    [KYUUBI #4936] [KYUUBI#4935][Improvement] More than target num of executors 
may survive after FinalStageResourceManager did kill
    
    ### _Why are the changes needed?_
    When FinalStageResourceManager chooses executors to be killed, it may add 
dead executors to the kill list.
    This will leave more than target num of executors survived and cause 
resource waste.
    
    ### _How was this patch tested?_
    - [ ] Add some test cases that check the changes thoroughly including 
negative and positive cases if possible
    
    - [ ] Add screenshots for manual tests if appropriate
    
    - [x] [Run 
test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests)
 locally before make a pull request
    
    Closes #4936 from zhouyifan279/kill-executor.
    
    Closes #4936
    
    2aaa84cb1 [zhouyifan279] [KYUUBI#4935][Improvement] More than target num of 
executors may survive after FinalStageResourceManager did kill
    
    Authored-by: zhouyifan279 <zhouyifan...@gmail.com>
    Signed-off-by: Cheng Pan <cheng...@apache.org>
---
 .../src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/extensions/spark/kyuubi-extension-spark-3-3/src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala
 
b/extensions/spark/kyuubi-extension-spark-3-3/src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala
index 7a0ae1592..dc573f838 100644
--- 
a/extensions/spark/kyuubi-extension-spark-3-3/src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala
+++ 
b/extensions/spark/kyuubi-extension-spark-3-3/src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala
@@ -170,7 +170,7 @@ case class FinalStageResourceManager(session: SparkSession)
 
     // Evict the rest executors according to the shuffle block size
     executorToBlockSize.toSeq.sortBy(_._2).foreach { case (id, _) =>
-      if (executorIdsToKill.length < expectedNumExecutorToKill) {
+      if (executorIdsToKill.length < expectedNumExecutorToKill && 
existedExecutors.contains(id)) {
         executorIdsToKill.append(id)
       }
     }