GitHub user tgravescs opened a pull request:
https://github.com/apache/spark/pull/19849
[SPARK-22653] executorAddress registered in
CoarseGrainedSchedulerBackend.executorDataMap is null
https://issues.apache.org/jira/browse/SPARK-22653
executorRef.address can be null, pass the executorAddress which accounts
for it being null a few lines above the fix.
Manually tested this patch. You can reproduce the issue by running a
simple spark-shell in yarn client mode with dynamic allocation and request some
executors up front. Let those executors idle timeout. Get a heap dump.
Without this fix, you will see that addressToExecutorId still contains the ids,
with the fix addressToExecutorId is properly cleaned up.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tgravescs/spark SPARK-22653
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19849.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19849
----
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
