[
https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14385890#comment-14385890
]
Steve Loughran commented on SPARK-1537:
---------------------------------------
# I've just tried to see where YARN-2444 stands; I can't replicate it in trunk
but I've submitted the tests to verify that it isn't there.
# for YARN-2423 Spark seems kind of trapped. It needs an api tagged as
public/stable; Robert's patch has the API, except it's being rejected on the
basis that "ATSv2 will break it". So it can't be tagged as stable. So there's
no API for GET operations until some undefined time {{t1 > now()}} —and then,
only for Hadoop versions with it. Which implies it won't get picked up by Spark
for a long time.
I think we need to talk to the YARN dev team and see what can be done here.
Even if there's no API client bundled into YARN, unless the v1 API and its
paths beginning {{/ws/v1/timeline/}} are going to go away, then a REST client
is possible; it may just have to be done spark-side, where at least it can be
made resilient to hadoop versions.
> Add integration with Yarn's Application Timeline Server
> -------------------------------------------------------
>
> Key: SPARK-1537
> URL: https://issues.apache.org/jira/browse/SPARK-1537
> Project: Spark
> Issue Type: New Feature
> Components: YARN
> Reporter: Marcelo Vanzin
> Assignee: Marcelo Vanzin
> Attachments: SPARK-1537.txt, spark-1573.patch
>
>
> It would be nice to have Spark integrate with Yarn's Application Timeline
> Server (see YARN-321, YARN-1530). This would allow users running Spark on
> Yarn to have a single place to go for all their history needs, and avoid
> having to manage a separate service (Spark's built-in server).
> At the moment, there's a working version of the ATS in the Hadoop 2.4 branch,
> although there is still some ongoing work. But the basics are there, and I
> wouldn't expect them to change (much) at this point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
