(spark) branch branch-3.5 updated: [SPARK-46006][YARN][FOLLOWUP] YarnAllocator set target executor number to 0 to cancel pending allocate request when driver stop

[yao]

This is an automated email from the ASF dual-hosted git repository.

yao pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.5 by this push:
     new 92b6619d3ffe [SPARK-46006][YARN][FOLLOWUP] YarnAllocator set target 
executor number to 0 to cancel pending allocate request when driver stop
92b6619d3ffe is described below

commit 92b6619d3ffe3531ac7b11363bf68ad4a6cc8f1e
Author: Angerszhuuuu <angers....@gmail.com>
AuthorDate: Tue Nov 28 11:04:14 2023 +0800

    [SPARK-46006][YARN][FOLLOWUP] YarnAllocator set target executor number to 0 
to cancel pending allocate request when driver stop
    
    ### What changes were proposed in this pull request?
    YarnAllocator set target executor number to 0 to cancel pending allocate 
request when driver stop
    Now for this issue we do:
    
    1. AllocationFailure should not be treated as exitCausedByApp when driver 
is shutting down https://github.com/apache/spark/pull/38622
    2. Avoid new allocation requests when sc.stop stuck 
https://github.com/apache/spark/pull/43906
    3. Cancel pending allocation request, this pr 
https://github.com/apache/spark/pull/44036
    
    ### Why are the changes needed?
    Avoid unnecessary allocate request
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    MT
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #44036 from AngersZhuuuu/SPARK-46006-FOLLOWUP.
    
    Authored-by: Angerszhuuuu <angers....@gmail.com>
    Signed-off-by: Kent Yao <y...@apache.org>
    (cherry picked from commit dbc8756bdac823be42ed10bc011415f405905497)
    Signed-off-by: Kent Yao <y...@apache.org>
---
 .../src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala  | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git 
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
 
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
index f14fc9d5de46..5fccc8c9ff47 100644
--- 
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
+++ 
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
@@ -385,7 +385,10 @@ private[yarn] class YarnAllocator(
     this.hostToLocalTaskCountPerResourceProfileId = 
hostToLocalTaskCountPerResourceProfileId
 
     if (resourceProfileToTotalExecs.isEmpty) {
-      targetNumExecutorsPerResourceProfileId.clear()
+      // Set target executor number to 0 to cancel pending allocate request.
+      targetNumExecutorsPerResourceProfileId.keys.foreach { rp =>
+        targetNumExecutorsPerResourceProfileId(rp) = 0
+      }
       allocatorNodeHealthTracker.setSchedulerExcludedNodes(excludedNodes)
       true
     } else {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org