[
https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190946#comment-14190946
]
Zhijie Shen commented on SPARK-1537:
------------------------------------
bq. That's exactly my point about the ATS not being production-level quality
yet. The current plans I'm aware of would require changes in the ATS API.
Not to mention the definition of production ready (which differs from community
to community, such as Tez and MapReduce), I'm curious about the required API
changes of the timeline server. Please elaborate the *changes* in case I've
missed some discussion. On the other side, according to my understanding of the
timeline server, the ongoing and the future improvement is:
1) Security is coming with Hadoop 2.6, which doesn't affect the usage of the
existing APIs in a insecure mode. AFAIK, Spark is working with Hadoop 2.3(4).
It should be okay to ride on the timeline server in insecure mode. Whenever
upgrading to Hadoop 2.6, you just need to turn on the security switch.
2) Timeline availability and scalability is going to be a server side
improvement, but doesn't affect user-faced API. In the scope of YARN, we have
already successfully enhance RM with the HA feature while making it transparent
to the user. I'm not aware of the major blocker that prevents the timeline
server to achieve the same goal.
3) For the client libs, we're trying to help to users to utilize the timeline
service more easily (e.g., YARN-2517, YARN-2673), which are either transparent
or additions. As I've mentioned before, we're careful about any proposed
changes that will break the incompatibility.
I'm commenting on this Jira to share more insights about the timeline server to
Spark folks in case the folks interested in this YARN offer. It's up to Spark
folks to decide whether they want to make use of it or when they make use of it.
> Add integration with Yarn's Application Timeline Server
> -------------------------------------------------------
>
> Key: SPARK-1537
> URL: https://issues.apache.org/jira/browse/SPARK-1537
> Project: Spark
> Issue Type: New Feature
> Components: YARN
> Reporter: Marcelo Vanzin
> Assignee: Marcelo Vanzin
>
> It would be nice to have Spark integrate with Yarn's Application Timeline
> Server (see YARN-321, YARN-1530). This would allow users running Spark on
> Yarn to have a single place to go for all their history needs, and avoid
> having to manage a separate service (Spark's built-in server).
> At the moment, there's a working version of the ATS in the Hadoop 2.4 branch,
> although there is still some ongoing work. But the basics are there, and I
> wouldn't expect them to change (much) at this point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
