[
https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306552#comment-14306552
]
Andrew Or commented on SPARK-5388:
----------------------------------
Hi [~vanzin], thank you for all of your comments. I agree with many of the
points that you raised and have addressed them in the PR. I'd just like to
clarify, in case there is any confusion, that this is intended to be an
internal submission gateway only for standalone cluster mode. This is in no way
expected to interface with third party applications or intended to be a stable
alternative to Spark submit across all cluster managers and deploy modes
For this reason, the protocol has very few core requirements, the most
important of which, backward and forward compatibility, is sufficiently handled
by using any stable RPC framework. It's just so that we chose a HTTP/JSON-based
design because it is the simplest one that fulfills this requirement. The
initial design did not adhere closely to conventions commonly used by other
REST protocols because this was not strictly necessary to achieve the
compatibility guarantees we wanted to provide. Not to mention there is not
really a formal W3C-style REST specification, and many existing REST protocols
out there (e.g. twitter, flickr, github) actually diverge widely in terms of
the extent to which they adherence to these vaguely-defined conventions.
That said, I agree completely that the initial design doc was vague. I have
updated it by expanding in much greater detail on things like how the server
responds to different error conditions and how versioning is handled. Please
have a look and let me know if you have any other concerns.
> Provide a stable application submission gateway in standalone cluster mode
> --------------------------------------------------------------------------
>
> Key: SPARK-5388
> URL: https://issues.apache.org/jira/browse/SPARK-5388
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Blocker
> Attachments: stable-spark-submit-in-standalone-mode-2-4-15.pdf
>
>
> The existing submission gateway in standalone mode is not compatible across
> Spark versions. If you have a newer version of Spark submitting to an older
> version of the standalone Master, it is currently not guaranteed to work. The
> goal is to provide a stable REST interface to replace this channel.
> The first cut implementation will target standalone cluster mode because
> there are very few messages exchanged. The design, however, should be general
> enough to potentially support this for other cluster managers too. Note that
> this is not necessarily required in YARN because we already use YARN's stable
> interface to submit applications there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
