Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
@mccheah Tried this PR in client-mode In-Cluster on minikube v0.25.2:
Exectuors are started but directly removed. As the start/remove is so fast, I
can hardly see logs (and the logs I have seen don't show any stacktrace). Maybe
something in my env?
The config I have for the client mode is:
```
# DRIVER_POD_NAME=$HOSTNAME
--conf spark.kubernetes.driver.pod.name="$DRIVER_POD_NAME" \
--conf spark.driver.host="$DRIVER_POD_NAME" \
--conf spark.driver.port=7077 \
--conf spark.driver.blockManager.port=10000 \
```
The driver log is:
```
2018-07-18 14:29:43 INFO SparkContext:54 - Created broadcast 0 from
broadcast at DAGScheduler.scala:1039
2018-07-18 14:29:43 INFO DAGScheduler:54 - Submitting 10 missing tasks
from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15
tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
2018-07-18 14:29:43 INFO TaskSchedulerImpl:54 - Adding task set 0.0 with
10 tasks
2018-07-18 14:29:45 INFO ExecutorPodsAllocator:54 - Going to request 1
executors from Kubernetes.
2018-07-18 14:29:46 INFO BlockManagerMasterEndpoint:54 - Trying to remove
executor 5 from BlockManagerMaster.
2018-07-18 14:29:46 INFO BlockManagerMaster:54 - Removal of executor 5
requested
2018-07-18 14:29:46 INFO
KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove
non-existent executor 5
2018-07-18 14:29:52 INFO BlockManagerMaster:54 - Removal of executor 6
requested
2018-07-18 14:29:52 INFO
KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove
non-existent executor 6
2018-07-18 14:29:52 INFO BlockManagerMasterEndpoint:54 - Trying to remove
executor 6 from BlockManagerMaster.
2018-07-18 14:29:52 INFO ExecutorPodsAllocator:54 - Going to request 1
executors from Kubernetes.
2018-07-18 14:29:55 INFO BlockManagerMaster:54 - Removal of executor 7
requested
2018-07-18 14:29:55 INFO
KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove
non-existent executor 7
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
