[
https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14304010#comment-14304010
]
Patrick Wendell commented on SPARK-5388:
----------------------------------------
The intention for this is really just to take single RPC that was using Akka
and add a stable version of it that we are okay supporting long term. It
doesn't preclude moving to avro or some other RPC as a general thing we use
across all of Spark. However, that design choice was intentionally excluded
from this decision given all the complexities you bring up. Doing some basic
message dispatching on our own - there is only a small and very straightforward
code related to this. Adopting Avro would be overkill for this.
In the current implementation the client and server exchange Spark versions, so
this is the basis of reasoning about version changes - maybe it wasn't in the
design doc. In terms of evolvability, the way you do this is that you only add
new functionality over time, and you never remove fields from messages. This is
similar to the API contract of the history logs with the history server. So the
idea is that newer clients would implement a super set of messages and fields
as older ones.
Adding v1 seems like a good idea in case this evolves into something public or
more well specified over time. It would just be good to define precisely what
it means to advance that version identifier. That all matters a lot more if we
want it to be something others interact with.
> Provide a stable application submission gateway in standalone cluster mode
> --------------------------------------------------------------------------
>
> Key: SPARK-5388
> URL: https://issues.apache.org/jira/browse/SPARK-5388
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Blocker
> Attachments: Stable Spark Standalone Submission.pdf
>
>
> The existing submission gateway in standalone mode is not compatible across
> Spark versions. If you have a newer version of Spark submitting to an older
> version of the standalone Master, it is currently not guaranteed to work. The
> goal is to provide a stable REST interface to replace this channel.
> The first cut implementation will target standalone cluster mode because
> there are very few messages exchanged. The design, however, should be general
> enough to potentially support this for other cluster managers too. Note that
> this is not necessarily required in YARN because we already use YARN's stable
> interface to submit applications there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
