Do you have event logging enabled? That could be the problem. The Master tries to aggressively recreate the web ui of the completed job with the event logs (when it is enabled) causing the Master to stall. I created a JIRA for this. https://issues.apache.org/jira/browse/SPARK-6270
On Tue, Mar 10, 2015 at 7:10 PM, Xuelin Cao <xuelincao2...@gmail.com> wrote: > > Hey, > > Recently, we found in our cluster, that when we kill a spark > streaming app, the whole cluster cannot response for 10 minutes. > > And, we investigate the master node, and found the master process > consumes 100% CPU when we kill the spark streaming app. > > How could it happen? Did anyone had the similar problem before? > > > > >
