GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/9182
SPARK-1537: pure service API and test service
This is purely the yarn/src/main and yarn/src/test bits of the YARN ATS
integration: the extension model to load and run implementations of
`SchedulerExtensionService` in the yarn cluster scheduler process âand to
stop them afterwards.
There's duplication between the two schedulers, yarn-client and
yarn-cluster, at least in terms of setting everything up, because the common
superclass, `YarnSchedulerBackend` is in spark-core, and the extension services
need the YARN app/attempt IDs.
If you look at how the the extension services are loaded, the case class
`SchedulerExtensionServiceBinding` is used to pass in config info -currently
just the spark context and the yarn IDs, of which one, the attemptID, will be
null when running client-side. I'm passing in a case class to ensure that it
would be possible in future to add extra arguments to the binding class, yet,
as the method signature will not have changed, still be able to load existing
services.
There's no functional extension service here, just one for testing. The
real tests come in the bigger pull requests. At the same time, there's no
restriction of this extension service purely to the ATS history publisher.
Anything else that wants to listen to the spark context and publish events
could use this, and I'd also consider writing one for the YARN-913 registry
service, so that the URLs of the web UI would be locatable through that (low
priority; would make more sense if integrated with a REST client).
There's no minicluster test. Given the test execution overhead of setting
up minicluster tests, it'd probably be better to add an extension service into
one of the existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/steveloughran/spark
stevel/feature/SPARK-1537-service
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9182.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9182
----
commit e796bb33d3128587f87529284fe45b37e1205de5
Author: Steve Loughran <[email protected]>
Date: 2015-10-20T14:35:40Z
SPARK-1537: pure service API and test service
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
