Probably the cleanup work like clean shuffle files, tmp files cost too much of CPUs, since if we run Spark Streaming for a long time, lots of files will be generated, so cleanup this files before app is exited could be time-consuming.
Thanks Jerry 2015-03-11 10:43 GMT+08:00 Tathagata Das <t...@databricks.com>: > Do you have event logging enabled? > That could be the problem. The Master tries to aggressively recreate the > web ui of the completed job with the event logs (when it is enabled) > causing the Master to stall. > I created a JIRA for this. > https://issues.apache.org/jira/browse/SPARK-6270 > > On Tue, Mar 10, 2015 at 7:10 PM, Xuelin Cao <xuelincao2...@gmail.com> > wrote: > >> >> Hey, >> >> Recently, we found in our cluster, that when we kill a spark >> streaming app, the whole cluster cannot response for 10 minutes. >> >> And, we investigate the master node, and found the master process >> consumes 100% CPU when we kill the spark streaming app. >> >> How could it happen? Did anyone had the similar problem before? >> >> >> >> >> >
