+1 for JSON.  The job server could also save context history and state and
make it queryable via REST APIs regardless of YARN / Mesos / standalone.


On Tue, Jan 7, 2014 at 2:51 PM, Tom Graves <tgraves...@yahoo.com> wrote:

> I don't think you want to save the html/xml files. I would rather see the
> info saved into a history file in like a json format that could then be
> re-read and the web ui display the info, hopefully without much change to
> the UI parts.  For instance perhaps the history server could read the file
> and populate the appropriate Spark data structures that the web ui already
> uses.
>
> I would suggest making it so the history server is an optional server and
> could be run on any node. That way if the load on a particular node becomes
> to much it could be moved, but you also could run it on the same node as
> the Master.  All it really needs to know is where to get the history files
> from and have access to that location.
>
> Hadoop actually has a history server for MapReduce which works very
> similar to what I mention above.   One thing to keep in minds here is
> security.  You want to make sure that the history files can only be read by
> users who have the appropriate permissions.  The history server itself
> could run as  a superuser who has permission to server up the files based
> on the acls.
>
>
>
> On Tuesday, January 7, 2014 8:06 AM, "Xia, Junluan" <junluan....@intel.com>
> wrote:
>
> Hi all
>          Spark job web ui will not be available when job is over, but it
> is convenient for developer to debug with persisting job web ui. I just
> come up with draft for this issue.
>
> 1.       We could simply save the web page with html/xml
> format(stages/executors/storages/environment) to certain location when job
> finished
>
> 2.       But it is not easy for user to review the job info with #1, we
> could build extra job history service for developers
>
> 3.       But where will we build this history service? In Driver node or
> Master node?
>
> Any suggestions about this improvement?
>
> regards,
> Andrew
>



-- 
--
Evan Chan
Staff Engineer
e...@ooyala.com  |

<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>

Reply via email to