Github user caneGuy commented on the issue:
https://github.com/apache/spark/pull/19132
@jerryshao Thanks for your time. IIUC, event log is completed since driver
has not dropped any event of executor which has problem described above.See
below,driver only drop two events after shutting down:
`2017-09-11,17:08:57,605 ERROR org.apache.spark.scheduler.LiveListenerBus:
SparkListenerBus has already stopped! Dropping event
SparkListenerExecutorMetricsUpdate(300,WrappedArray())
2017-09-11,17:08:57,623 INFO org.apache.spark.deploy.yarn.YarnAllocator:
Driver requested a total number of 0 executor(s).
2017-09-11,17:08:57,624 INFO
org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Shutting down
all executors
2017-09-11,17:08:57,624 INFO
org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Asking each
executor to shut down
2017-09-11,17:08:57,629 ERROR org.apache.spark.scheduler.LiveListenerBus:
SparkListenerBus has already stopped! Dropping event
SparkListenerExecutorMetricsUpdate(258,WrappedArray())`
And below showing an executor has this problem:
Case here is a job running today of our cluster.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
