Hi, I don't have a complete answer to your questions but:
"Removing the suffix does not solve the problem" -> unfortunately this is true, the master web UI only tries to build out a Spark UI from the event logs once, at the time the context is closed. If the event logs are in-progress at this time, then you basically missed the opportunity. "Does it mean I don't need to start history server if I only use spark in standalone mode?" -> Yes, you don't need to start the history server. On Mon, Jun 8, 2015 at 7:57 PM, Du Li <l...@yahoo-inc.com.invalid> wrote: > Event log is enabled in my spark streaming app. My code runs in standalone > mode and the spark version is 1.3.1. I periodically stop and restart the > streaming context by calling ssc.stop(). However, from the web UI, when > clicking on a past job, it says the job is still in progress and does not > show the event log. The event log files have suffix .inprogress. Removing > the suffix does not solve the problem. Do I need to do anything here in > order to view the event logs of finished jobs? Or do I need to stop ssc > differently? > > In addition, the documentation seems to suggest history server is used for > Mesos or YARN mode. Does it mean I don't need to start history server if I > only use spark in standalone mode? > > Thanks, > Du >
