GitHub user devaraj-kavali opened a pull request:

    https://github.com/apache/spark/pull/20754

    [SPARK-23287][MESOS] Spark scheduler does not remove initial executor if 
not one job submitted

    ## What changes were proposed in this pull request?
    
    In `ExecutorAllocationManager.schedule()`, `numExecutorsTarget` is getting 
updated as part of `updateAndSyncNumExecutorsTarget(now)` but it skips updating 
till initializing becomes false, `removeExecutors()` is not removing the 
expired executors since the condition `else if (newExecutorTotal - 1 < 
numExecutorsTarget) { ` is satisfying to skip them, and they are missing to 
remove and continues running till the application completes.
    
    I moved the `updateAndSyncNumExecutorsTarget(now)` to after the expiry 
check and initializing var assignment if eligible so that the updated 
`numExecutorsTarget `can be used while removing executors. 
    
    ## How was this patch tested?
    
    I verified it manually by enabling the dynamic allocation with Mesos mode, 
now it removes the executors when they are not getting assigned any task for 
the specified executorIdleTimeout. 


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/devaraj-kavali/spark SPARK-23287

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20754.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20754
    
----
commit 5bef384acfe3d76949dceab669743e15373bad57
Author: Devaraj K <devaraj@...>
Date:   2018-03-07T01:54:58Z

    SPARK-23287 Spark scheduler does not remove initial executor if not one
    job submitted

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to