[
https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302006#comment-14302006
]
Patrick Wendell commented on SPARK-5388:
----------------------------------------
Hey [~vanzin] and [~tigerquoll], the original goal of this was just to provide
a stable API between the spark-submit and the standalone master, similar to
what YARN has. And a simple JSON/REST API was something that is easy to support
in the long term.
In terms of using more of the standard web services conventions for the HTTP
method (i.e. using GET, POST, and DELETE), I agree that would be nicer if it's
not too hard to do. It be mostly for clarity though rather than correctness,
this isn't something where we expect things like caching of responses, etc,
which depend on strict interpretation of the HTTP method.
In terms of versioning, the original proposal has the client and server
communicate versions, so the idea is that compatibility could be handled on
that basis, and that all older versions would be supported (hence this being a
stable API). For web services API's they sometimes have "v1", "v2" labels at
the URL path. However, this type of versioning implies breakage of older
versions, which is exactly what we want to avoid, right? For instance if github
adds new fields to a return type (in a way that is backward compatible), I'm
assuming they don't increment the version identifier, right? I think the idea
here is similar to our handling of the event logs, we want to make sure we
support older versions by only increasing what is communicated over time, and
having the server gracefully handle older clients. So I'm not totally sure what
the "v2" style labeling would help with unless we actually think we are going
to break this API in a way that's not backwards compatible.
> Provide a stable application submission gateway in standalone cluster mode
> --------------------------------------------------------------------------
>
> Key: SPARK-5388
> URL: https://issues.apache.org/jira/browse/SPARK-5388
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Blocker
> Attachments: Stable Spark Standalone Submission.pdf
>
>
> The existing submission gateway in standalone mode is not compatible across
> Spark versions. If you have a newer version of Spark submitting to an older
> version of the standalone Master, it is currently not guaranteed to work. The
> goal is to provide a stable REST interface to replace this channel.
> The first cut implementation will target standalone cluster mode because
> there are very few messages exchanged. The design, however, should be general
> enough to potentially support this for other cluster managers too. Note that
> this is not necessarily required in YARN because we already use YARN's stable
> interface to submit applications there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
