liujianhui created SPARK-18994:
----------------------------------
Summary: worker clean up app directory block the heartbeat sending
Key: SPARK-18994
URL: https://issues.apache.org/jira/browse/SPARK-18994
Project: Spark
Issue Type: Bug
Components: Deploy
Affects Versions: 1.6.1
Reporter: liujianhui
h2. scene
# when an application stop, it will trigger master send ApplicationFinish event
to worker
# worker will clean up the directories for the application, but it takes a long
time to delete them, as a result of the worker endpoint is Thread safe, and
then it block the heartbeat sending to master
# master find the heartbeat expired for that worker, and then remove the
worker, it will re-launching the driver belong to that worker, actually that
driver doesn't exit, finally this will exist two app with same driver-id
the worker log
{code}
2016-12-23 11:39:33,033 |INFO | dispatcher-event-loop-26 |
org.apache.spark.deploy.worker.Worker | Executor app-20161208222600-0020/200
finished with state KILLED exitStatus 137 |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:39:33,033 |INFO | dispatcher-event-loop-26 |
org.apache.spark.deploy.worker.Worker | Cleaning up local directories for
application app-20161208222600-0020 |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver | Application
app-20161208222600-0020 removed, cleanupLocalDirs = true |
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.applicationRemoved(ExternalShuffleBlockResolver.java:206)
2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
org.apache.spark.deploy.worker.Worker | Master with url
spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-2 |
org.apache.spark.deploy.worker.Worker | Connecting to master
bigdata-sirius-ser02.gz01:8090... |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-3 |
org.apache.spark.deploy.worker.Worker | Connecting to master
bigdata-sirius-ser03.gz01:8090... |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
org.apache.spark.deploy.worker.Worker | Master with url
spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | shuffle-client-1 |
org.apache.spark.deploy.worker.Worker | Successfully registered with master
spark://bigdata-sirius-ser02.gz01:8090 |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | shuffle-client-1 |
org.apache.spark.deploy.worker.Worker | Worker cleanup enabled; old application
directories will be deleted in: /data1/spark/work |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
org.apache.spark.deploy.worker.Worker | Not spawning another attempt to
register with the master, since there is an attempt scheduled already. |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
org.apache.spark.deploy.worker.Worker | Master with url
spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
org.apache.spark.deploy.worker.Worker | Master with url
spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-3 |
org.apache.spark.deploy.worker.Worker | Connecting to master
bigdata-sirius-ser03.gz01:8090... |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 |
org.apache.spark.deploy.worker.Worker | Not spawning another attempt to
register with the master, since there is an attempt scheduled already. |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-2 |
org.apache.spark.deploy.worker.Worker | Connecting to master
bigdata-sirius-ser02.gz01:8090... |
org.apache.spark.Logging$class.logInfo(Logging.scala:58)
{code}
*master* log
{code}
2016-12-23 11:40:46,046 |WARN | dispatcher-event-loop-20 |
org.apache.spark.deploy.master.Master | Removing
worker-20161208233029-100.69.146.57-26874 because we got no heartbeat in 60
seconds | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker
worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
org.apache.spark.Logging$class.logWarning(Logging.scala:70)
2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker
worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
org.apache.spark.Logging$class.logWarning(Logging.scala:70)
2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker
worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
org.apache.spark.Logging$class.logWarning(Logging.scala:70)
2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker
worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
org.apache.spark.Logging$class.logWarning(Logging.scala:70)
2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker
worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
org.apache.spark.Logging$class.logWarning(Logging.scala:70)
2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker
worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
org.apache.spark.Logging$class.logWarning(Logging.scala:70)
2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 |
org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker
worker-20161208233029-100.69.146.57-26874. Asking it to re-register. |
org.apache.spark.Logging$class.logWarning(Logging.scala:70)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
