[
https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112791#comment-14112791
]
Matei Zaharia commented on SPARK-3215:
--------------------------------------
Hey Marcelo, while this could be useful for Spark, have you thought of trying
an application-level approach initially for Hive? The reason is that this is
pretty easy to do at the application level (it's more or less just RPC), and
different users might want to do RPC in different ways, so I'm not sure we need
to be in the business of dictating one way to run it. Something that would be
more useful for Spark, but also much harder to implement, is an interface that
lets you write jobs against the *current* Spark API (without modifying them)
but have the bulk of the SparkContext execute elsewhere.
If the issue is sending back metrics on the jobs, maybe we can have APIs to
enable you to send those at the application level.
> Add remote interface for SparkContext
> -------------------------------------
>
> Key: SPARK-3215
> URL: https://issues.apache.org/jira/browse/SPARK-3215
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Reporter: Marcelo Vanzin
> Labels: hive
> Attachments: RemoteSparkContext.pdf
>
>
> A quick description of the issue: as part of running Hive jobs on top of
> Spark, it's desirable to have a SparkContext that is running in the
> background and listening for job requests for a particular user session.
> Running multiple contexts in the same JVM is not a very good solution. Not
> only SparkContext currently has issues sharing the same JVM among multiple
> instances, but that turns the JVM running the contexts into a huge bottleneck
> in the system.
> So I'm proposing a solution where we have a SparkContext that is running in a
> separate process, and listening for requests from the client application via
> some RPC interface (most probably Akka).
> I'll attach a document shortly with the current proposal. Let's use this bug
> to discuss the proposal and any other suggestions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
