[
https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290847#comment-14290847
]
Dale Richardson commented on SPARK-5388:
----------------------------------------
Hi Andrew,
I think the idea is well worth considering. In response to the requirement of
making it easier for client<-> master communications to pass through
restrictive fire-walling, have you considered just using Akka's REST gateway
(http://doc.akka.io/docs/akka/1.3.1/scala/http.html)?
I also have a question if there is an intention for other entities (such as job
servers) to communicate with the master at all? If so then the proposed gateway
is semantically defined at a fairly low level (just RPC over JSON/HTTP). This
is fine if the interface is not going to be exposed to anybody who is not a
spark developer with detailed knowledge of spark internals. Did you use the
term “REST” to simply mean RPC over JSON/HTTP?
Creating a REST interface is more then a HTTP RPC gateway. If the interface is
going to be exposed to 3rd parties (such as developers of Job servers and web
notebooks etc) then there is a benefit to simplifying some of the exposed
application semantics, and exposing an API that is more integrated with HTTP’s
protocol semantics which most people are already familiar with - this is what a
true REST interface does and if you are defining an endpoint for others to use
it is a very powerful concept that allows other people to quickly grasp how to
properly use the exposed interface.
A rough sketch of a more “REST”ed version of the API would be:
*Submit_driver_request*
HTTP POST JSON body of request http://host:port/SparkMaster?SubmitDriver
Responds with standard HTTP Response including allocated DRIVER_ID if driver
submission ok, http error codes with spark specific error if not.
*Get status of DRIVER*
HTTP GET http://host:port/SparkMaster/Drivers/<DRIVER_ID>
Responds with JSON body containing information on driver execution. If no
record of driver_id, then http error code 404 (Not found) returned.
*Kill Driver request*
HTTP DELETE http://host:port/SparkMaster/Drivers/<DRIVER_ID>
Responds with JSON body containing information on driver kill request, or http
error code if an error occurs.
I would be happy to prototype something like this up to test the concept out
for you if you are looking for something more than just RPC over JSON/HTTP.
> Provide a stable application submission gateway
> -----------------------------------------------
>
> Key: SPARK-5388
> URL: https://issues.apache.org/jira/browse/SPARK-5388
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Blocker
> Attachments: Stable Spark Standalone Submission.pdf
>
>
> The existing submission gateway in standalone mode is not compatible across
> Spark versions. If you have a newer version of Spark submitting to an older
> version of the standalone Master, it is currently not guaranteed to work. The
> goal is to provide a stable REST interface to replace this channel.
> The first cut implementation will target standalone cluster mode because
> there are very few messages exchanged. The design, however, will be general
> enough to eventually support this for other cluster managers too. Note that
> this is not necessarily required in YARN because we already use YARN's stable
> interface to submit applications there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
