[ https://issues.apache.org/jira/browse/SPARK-19802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893580#comment-15893580 ]
Saisai Shao commented on SPARK-19802: ------------------------------------- Spark's {{ApplicationHistoryProvider}} is pluggable, user could implement their own provider and plug into Spark's history server. So you could implement a {{HistoryProvider}} you wanted out of Spark. >From your description, this is more like a Hadoop ATS (Hadoop application >timeline server). We have an implementation of Timeline based history provider >for Spark's history server. The main feature is like what you mentioned query >through TCP, get the event and display on UI. > Remote History Server > --------------------- > > Key: SPARK-19802 > URL: https://issues.apache.org/jira/browse/SPARK-19802 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.1.0 > Reporter: Ben Barnard > > Currently the history server expects to find history in a filesystem > somewhere. It would be nice to have a history server that listens for > application events on a TCP port, and have a EventLoggingListener that sends > events to the listening history server instead of writing to a file. This > would allow the history server to show up-to-date history for past and > running jobs in a cluster environment that lacks a shared filesystem. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org