+1 for JSON. The job server could also save context history and state and make it queryable via REST APIs regardless of YARN / Mesos / standalone.
On Tue, Jan 7, 2014 at 2:51 PM, Tom Graves <tgraves...@yahoo.com> wrote: > I don't think you want to save the html/xml files. I would rather see the > info saved into a history file in like a json format that could then be > re-read and the web ui display the info, hopefully without much change to > the UI parts. For instance perhaps the history server could read the file > and populate the appropriate Spark data structures that the web ui already > uses. > > I would suggest making it so the history server is an optional server and > could be run on any node. That way if the load on a particular node becomes > to much it could be moved, but you also could run it on the same node as > the Master. All it really needs to know is where to get the history files > from and have access to that location. > > Hadoop actually has a history server for MapReduce which works very > similar to what I mention above. One thing to keep in minds here is > security. You want to make sure that the history files can only be read by > users who have the appropriate permissions. The history server itself > could run as a superuser who has permission to server up the files based > on the acls. > > > > On Tuesday, January 7, 2014 8:06 AM, "Xia, Junluan" <junluan....@intel.com> > wrote: > > Hi all > Spark job web ui will not be available when job is over, but it > is convenient for developer to debug with persisting job web ui. I just > come up with draft for this issue. > > 1. We could simply save the web page with html/xml > format(stages/executors/storages/environment) to certain location when job > finished > > 2. But it is not easy for user to review the job info with #1, we > could build extra job history service for developers > > 3. But where will we build this history service? In Driver node or > Master node? > > Any suggestions about this improvement? > > regards, > Andrew > -- -- Evan Chan Staff Engineer e...@ooyala.com | <http://www.ooyala.com/> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>