[
https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112943#comment-14112943
]
Matei Zaharia commented on SPARK-3215:
--------------------------------------
I think we should try this externally first and then see whether it makes sense
to put it in Spark. My reason is that as I said above, it's not clear that we
can get the RPC protocol and API right for all users. There's a lot of
complexity with RPC: threading model, the actual wire format chosen, the way
that interacts with upgrades (e.g. Protobuf is a nightmare and Thrift isn't
foolproof either), etc. In particular I see a lot of Spark client apps that are
also RPC servers or are communicating with other RPC systems, and it can be
tricky to mix two systems in the same app.
More generally though it would be awesome if you guys could use the Ooyala Job
Server for this, so that's a path to pursue. And again as with that, it may
make sense to move it into Spark eventually. We just have to make sure that
there are tangible benefits for it and that it's something we want to commit to
support long-term.
> Add remote interface for SparkContext
> -------------------------------------
>
> Key: SPARK-3215
> URL: https://issues.apache.org/jira/browse/SPARK-3215
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Reporter: Marcelo Vanzin
> Labels: hive
> Attachments: RemoteSparkContext.pdf
>
>
> A quick description of the issue: as part of running Hive jobs on top of
> Spark, it's desirable to have a SparkContext that is running in the
> background and listening for job requests for a particular user session.
> Running multiple contexts in the same JVM is not a very good solution. Not
> only SparkContext currently has issues sharing the same JVM among multiple
> instances, but that turns the JVM running the contexts into a huge bottleneck
> in the system.
> So I'm proposing a solution where we have a SparkContext that is running in a
> separate process, and listening for requests from the client application via
> some RPC interface (most probably Akka).
> I'll attach a document shortly with the current proposal. Let's use this bug
> to discuss the proposal and any other suggestions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
