[
https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14534810#comment-14534810
]
Steve Loughran commented on SPARK-1537:
---------------------------------------
For people who've not been tracking the WiP
# the timeline API is pretty thoroughly documented with examples; very close to
going in
[Latest
TimelineServer.md|https://github.com/steveloughran/hadoop-trunk/blob/stevel/YARN-3539-ATS-compatibility/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md]
# the timeline server integration is in sync with trunk, especially the
SPARK-4705 changes
# it has lots of tests. This includes: generating events from a spark context
and verifying that they are served up by an an-VM timeline server instance, and
retrievable by a REST client, bringing up a Spark History server and making GET
requests against it to verifying it hooks up to the server, and other
cross-system tests. That's about as much as you can do in a standalone unit
test suite.
# those tests all run happily on unix and windows, provided you set the
{{-Phadoop-2.6 -Pyarn}} flags to request a Hadoop 2.6 profile.
# and I've tested against hadoop 2.6.0, 2.7.0 & branch-2; everything compiles
and runs
Can I get some reviews?
> Add integration with Yarn's Application Timeline Server
> -------------------------------------------------------
>
> Key: SPARK-1537
> URL: https://issues.apache.org/jira/browse/SPARK-1537
> Project: Spark
> Issue Type: New Feature
> Components: YARN
> Reporter: Marcelo Vanzin
> Attachments: SPARK-1537.txt, spark-1573.patch
>
>
> It would be nice to have Spark integrate with Yarn's Application Timeline
> Server (see YARN-321, YARN-1530). This would allow users running Spark on
> Yarn to have a single place to go for all their history needs, and avoid
> having to manage a separate service (Spark's built-in server).
> At the moment, there's a working version of the ATS in the Hadoop 2.4 branch,
> although there is still some ongoing work. But the basics are there, and I
> wouldn't expect them to change (much) at this point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
