Probably the cleanup work like clean shuffle files, tmp files cost too much
of CPUs, since if we run Spark Streaming for a long time, lots of files
will be generated, so cleanup this files before app is exited could be
time-consuming.
Thanks
Jerry
2015-03-11 10:43 GMT+08:00 Tathagata Das t...@databricks.com:
Do you have event logging enabled?
That could be the problem. The Master tries to aggressively recreate the
web ui of the completed job with the event logs (when it is enabled)
causing the Master to stall.
I created a JIRA for this.
https://issues.apache.org/jira/browse/SPARK-6270
On Tue, Mar 10, 2015 at 7:10 PM, Xuelin Cao xuelincao2...@gmail.com
wrote:
Hey,
Recently, we found in our cluster, that when we kill a spark
streaming app, the whole cluster cannot response for 10 minutes.
And, we investigate the master node, and found the master process
consumes 100% CPU when we kill the spark streaming app.
How could it happen? Did anyone had the similar problem before?
