GitHub user tgravescs opened a pull request:
https://github.com/apache/spark/pull/19850
[SPARK-22653] executorAddress registered in CoarseGrainedSchedulerBac…
https://issues.apache.org/jira/browse/SPARK-22653
executorRef.address can be null, pass the executorAddress which accounts
for it being null a few lines above the fix.
Manually tested this patch. You can reproduce the issue by running a simple
spark-shell in yarn client mode with dynamic allocation and request some
executors up front. Let those executors idle timeout. Get a heap dump. Without
this fix, you will see that addressToExecutorId still contains the ids, with
the fix addressToExecutorId is properly cleaned up.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tgravescs/spark SPARK-22653
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19850.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19850
----
commit 2703214015c71ab4b2f7a0fba258b1107411e449
Author: Thomas Graves <[email protected]>
Date: 2017-11-29T16:43:10Z
[SPARK-22653] executorAddress registered in
CoarseGrainedSchedulerBackend.executorDataMap is null
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
