[
https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296068#comment-14296068
]
Marcelo Vanzin commented on SPARK-5388:
---------------------------------------
Hi [~andrewor14],
I read through the spec and the protocol specification seems to be lacking some
details. The mains things that bother me are:
- It's not really a REST API. There's a single endpoint to which you POST
different messages. This sort of forces your hand to use a custom
implementation, instead of being able to use a much nicer framework for this
purpose such as JAX-RS. Using a framework like that can later benefit other
parts of Spark too, such as providing a REST API for application data through
the web ui / history server. And as I mentioned in the PR, it allows you to
define the endpoints using classes or interfaces, which serves two purposes: it
allows you to do backwards compatibility checks with tools like MIMA, and it
allows you to use the client functionality of JAX-RS for client requests too
(and similar tools for other languages for those who, sort of feeding back into
Dale's comment). Plus, you can use things like Jackson and not care about how
to parse or generate JSON.
- It's unclear how the protocol will be allowed to evolve. What happens when
you add a new field or message in a later version, and that version tries to
submit to Spark 1.3? Is there a version negotiation up front, so that the new
client knows to use the old protocol if possible, or does the client just send
the new message and the server will complain if it contains things it doesn't
understand?
The latter kinda feeds into the first comment. With a proper REST-based API,
you'd put the first version of the protocol under "/v1", for example. Later
versions are added under "/v2" and can add new things. Client and server can
then negotiate up front (e.g, client needs at least version "x" for the current
app, asks the server for its supported versions, and complains if "x" is not
there).
Also, it could be more specific about how errors are reported. Do you get
specific HTTP error codes for different things? Is there an "Error" type that
is sent back to the client in JSON, and if so, what fields does it have?
> Provide a stable application submission gateway
> -----------------------------------------------
>
> Key: SPARK-5388
> URL: https://issues.apache.org/jira/browse/SPARK-5388
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Blocker
> Attachments: Stable Spark Standalone Submission.pdf
>
>
> The existing submission gateway in standalone mode is not compatible across
> Spark versions. If you have a newer version of Spark submitting to an older
> version of the standalone Master, it is currently not guaranteed to work. The
> goal is to provide a stable REST interface to replace this channel.
> The first cut implementation will target standalone cluster mode because
> there are very few messages exchanged. The design, however, will be general
> enough to eventually support this for other cluster managers too. Note that
> this is not necessarily required in YARN because we already use YARN's stable
> interface to submit applications there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
