Thomas Graves created SPARK-22653:
-------------------------------------
Summary: executorAddress registered in
CoarseGrainedSchedulerBackend.executorDataMap is null
Key: SPARK-22653
URL: https://issues.apache.org/jira/browse/SPARK-22653
Project: Spark
Issue Type: Bug
Components: Scheduler
Affects Versions: 2.2.0
Reporter: Thomas Graves
In CoarseGrainedSchedulerBackend.RegisterExecutor the executor data address
(executorRef.address) can be null.
val data = new ExecutorData(executorRef, executorRef.address, hostname,
cores, cores, logUrls)
At this point the executorRef.address can be null, there is actually code above
it that handles this case:
// If the executor's rpc env is not listening for incoming connections,
`hostPort`
// will be null, and the client connection should be used to contact
the executor.
val executorAddress = if (executorRef.address != null) {
executorRef.address
} else {
context.senderAddress
}
But it doesn't use executorAddress when it creates the ExecutorData.
This causes removeExecutor to never remove it properly from addressToExecutorId.
addressToExecutorId -= executorInfo.executorAddress
This is also a memory leak and can also call onDisconnected to call
disableExecutor when it shouldn't.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
