[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127564#comment-14127564 ] Marcelo Vanzin commented on SPARK-3215: --- For those following, I moved the prototype to this location: https://github.com/vanzin/spark-client This is so the Hive-on-Spark project can start playing with while we work on all the details. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123749#comment-14123749 ] Marcelo Vanzin commented on SPARK-3215: --- I updated the prototype to include a Java API and not to use SparkConf in the API. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14120657#comment-14120657 ] Matei Zaharia commented on SPARK-3215: -- Thanks Marcelo! Just a few notes on the API: - It needs to be Java-friendly, so it's probably not good to use Scala functions (e.g. `JobContext = T`) and maybe even the Future type. - It's a little weird to be passing a SparkConf to the JobClient, since most flags in there will not affect the jobs run (as they use the remote Spark cluster's SparkConf). Maybe it would be better to just pass a cluster URL. - It would be good to give jobs some kind of ID that client apps can log and can refer to even if the client crashes and the JobHandle object is gone. This is similar to how Hive prints the MapReduce job IDs it launched, and lets you kill them later using MR's hadoop job -kill. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14120708#comment-14120708 ] Marcelo Vanzin commented on SPARK-3215: --- Thanks Matei. Looking at a Java API is next in my TODO list - I want to look at how the RDD API does things and try to mimic that. I chose to use SparkConf because that's sort of standard; you may want to configure other things than just the cluster URL - e.g., executor count and size and things like that. So I wanted to avoid having to create yet another config object. It's a little unfortunate that SparkConf will inherit system properties, but since theoretically the application using the client is not a Spark application, it won't be using system properties to set SparkConf options. Also note that all options in the passed SparkConf instance are actually passed to the Spark app - so you will not be using the spark-defaults.conf file from anywhere. The code already creates a job ID internally, I just didn't expose it. That should be pretty simple to do. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118960#comment-14118960 ] Marcelo Vanzin commented on SPARK-3215: --- For those who'd prefer to see some code, here's a proof-of-concept: https://github.com/vanzin/spark/tree/SPARK-3215/remote Please ignore the fact that it's a module inside Spark; I picked a different package name so that I didn't end up using any internal Spark APIs. I just wanted to avoid having to write build code. In particular, focus on this package (and *not* what's inside impl): https://github.com/vanzin/spark/tree/SPARK-3215/remote/src/main/scala/org/apache/spark_remote That's all a user would see; what happens inside impl does not matter to the user. If you really want to look at the implementation code, it's currently using akka and has very little error handling. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112769#comment-14112769 ] Reynold Xin commented on SPARK-3215: I looked at the document. The high level proposal looks good. Can you update the document to include more details? In particular, a few things that are important to define are: 1. Interface for Future 2. Full interface for RemoteClient, including how to initialize it for different cluster mgr backends, how to add application jars 3. The RPC protocol between client/server: transport protocol, frameworks to use, version compatibility, etc 4. Project organization: should this module be in core, or should we create a new module in Spark (e.g. SparkContextClient module?) Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112780#comment-14112780 ] Marcelo Vanzin commented on SPARK-3215: --- Hi Reynold, thanks for the comments. This definitely needs more details, but I wanted to get the high-level idea out there first, since I've been told that in the past similar projects met some resistance. If everybody is ok with the approach, I'll go forward and write a proper spec. (I'm also working on a p.o.c. to test some ideas and use as a way to play with what the API would look like.) Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112791#comment-14112791 ] Matei Zaharia commented on SPARK-3215: -- Hey Marcelo, while this could be useful for Spark, have you thought of trying an application-level approach initially for Hive? The reason is that this is pretty easy to do at the application level (it's more or less just RPC), and different users might want to do RPC in different ways, so I'm not sure we need to be in the business of dictating one way to run it. Something that would be more useful for Spark, but also much harder to implement, is an interface that lets you write jobs against the *current* Spark API (without modifying them) but have the bulk of the SparkContext execute elsewhere. If the issue is sending back metrics on the jobs, maybe we can have APIs to enable you to send those at the application level. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[
https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112803#comment-14112803
]
Marcelo Vanzin commented on SPARK-3215:
---
Hi Matei,
Both suggestions came up during our internal talks. Doing it inside Hive is an
option, but we thought more people would benefit is this were a part of Spark.
While it may not be overly complicated, it's also not trivial, and having an
official and well-maintained way of doing it is an advantage of this
approach. It also keeps it closer to the core, making it easier to evolve to
accommodate new features in Spark.
I also toyed with the idea of use Spark API locally but have things run
remotely you mention, but I think that would require too many changes in Spark
to be useful. It also has some downsides for the client, which needs to run
more code / use more memory and thus suffers more from scalability and app
isolation issues.
As Reynold suggests, it doesn't need to be inside the {{core/}} project - it
could be a new sub-module.
Add remote interface for SparkContext
-
Key: SPARK-3215
URL: https://issues.apache.org/jira/browse/SPARK-3215
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Marcelo Vanzin
Labels: hive
Attachments: RemoteSparkContext.pdf
A quick description of the issue: as part of running Hive jobs on top of
Spark, it's desirable to have a SparkContext that is running in the
background and listening for job requests for a particular user session.
Running multiple contexts in the same JVM is not a very good solution. Not
only SparkContext currently has issues sharing the same JVM among multiple
instances, but that turns the JVM running the contexts into a huge bottleneck
in the system.
So I'm proposing a solution where we have a SparkContext that is running in a
separate process, and listening for requests from the client application via
some RPC interface (most probably Akka).
I'll attach a document shortly with the current proposal. Let's use this bug
to discuss the proposal and any other suggestions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112855#comment-14112855 ] Shivaram Venkataraman commented on SPARK-3215: -- This looks very interesting -- One thing that would be very useful is to make the RPC interface language agnostic. This would make it possible to submit Python or R jobs to a SparkContext without embedding a JVM in the driver process. Could we use Thrift or Protocol Buffers or something like that ? Also it'll be great to make a tentative list of RPCs that are required to get a simple application to work. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112931#comment-14112931 ] Evan Chan commented on SPARK-3215: -- [~vanzin] we should chat. I'm planning to move our Spark Job Server to have independent processes per SparkContext, and it already is Akka based. Something to consider is that you still need somebody to manage all the SparkContexts that you create, to make them HA, etc. So it is useful to have a layer like the job server. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112943#comment-14112943 ] Matei Zaharia commented on SPARK-3215: -- I think we should try this externally first and then see whether it makes sense to put it in Spark. My reason is that as I said above, it's not clear that we can get the RPC protocol and API right for all users. There's a lot of complexity with RPC: threading model, the actual wire format chosen, the way that interacts with upgrades (e.g. Protobuf is a nightmare and Thrift isn't foolproof either), etc. In particular I see a lot of Spark client apps that are also RPC servers or are communicating with other RPC systems, and it can be tricky to mix two systems in the same app. More generally though it would be awesome if you guys could use the Ooyala Job Server for this, so that's a path to pursue. And again as with that, it may make sense to move it into Spark eventually. We just have to make sure that there are tangible benefits for it and that it's something we want to commit to support long-term. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112958#comment-14112958 ] Marcelo Vanzin commented on SPARK-3215: --- I think the what RPC to use discussion is mostly irrelevant. It only matters if the API you want to expose is the RPC layer itself. My proposal is to expose a Scala/Java API, so how the bytes get from one side to the other underneath doesn't matter much. We can fiddle with that all we want as long as the Scala/Java API remains the same. Having the API at that level does make it trickier to support other languages, to address Shivaram's comments. But given what this feature proposes, I think it would be pretty hard to support multiple languages with a single backend anyway. If your client app is python, you need something on the server side that understands python. Evan, yes, there is code in the job server that does something similar to this; it's still sort of tied to the job server itself, and I actually don't think the job server part itself is very interesting - at least not for the Hive needs. Basically the remote context part of my proposal looks a lot like JobManagerActor, and you'd have a client library to talk to it directly (instead of going through the job server). It'd be interesting to know more about the changes you're working on, though. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113055#comment-14113055 ] Matei Zaharia commented on SPARK-3215: -- As I mentioned above, there's more to it than the Java / Scala API. First there's the threading model: how are you supposed to use this API within a multithreaded or asynchronous server? If the API is only blocking, it may be impossible to use it in some servers, and if it's non-blocking, you need the asynchronous callbacks to work in interoperable ways. Second there's the dependencies brought in. Anyway, we can see how this looks when complete. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113062#comment-14113062 ] Marcelo Vanzin commented on SPARK-3215: --- Hi Matei, sorry if I'm missing what you're trying to convey, but I don't see how any of your questions affect the choice of RPC. The (very) high-level API in the proposal is an async API based on futures, so a very well understood idiom. How you translate that into the underlying RPC layer, while important from an implementation perspective, is sort of irrelevant to the client using the API, in my view. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113121#comment-14113121 ] Matei Zaharia commented on SPARK-3215: -- The problem is just how different future or asynchrony libraries interact. Futures usually have a thread pool where callbacks run, some RPC libraries have their own thread pool, some people want to block an RPC call until a future completes, etc. When you combine a piece of software that uses one form of concurrency management (e.g. Akka or Jetty) with one that uses another (e.g. whatever we choose here), you can get problems. This is why for example Scala made Futures a part of the standard library (because people kept making separate Future libraries that were hard to combine). If there were good enough APIs in core for people to implement this kind of server manually, it would be easy for them to do it with their own framework. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113206#comment-14113206 ] Marcelo Vanzin commented on SPARK-3215: --- Matei, yes, all those things exist, but that is not what I'd like to discuss at this point. What I'm trying to discuss is: - Are these all the functional requirements needed to cover the use cases we have at hand (at the moment, Hive on Spark) - Is this something that should live in the core, alongside the core, or somewhere else entirely None of those depend on the choice of technology used to implement the feature, and none of those is affected by the things you mention. Those are all implementation details for when a consensus is reached over what to implement. Once those two questions are sorted out, yes, then we can start to discuss details of the API and how to implement it. But in my view it's too early to get into those discussions. And yes, as I said before, it's very possible for people to implement their own version of this. The point I'm making here is that it would be nice to have this readily available so people don't have to do that. Kinda like the Scala standard library added Futures so people would stop implementing their own... Regardless of the decision of where this code will live, it will have to exist, because Hive-on-Spark depends on it. We just thought it would be beneficial to Spark and its users to have it be generic enough to cover more than the Hive-on-Spark use case, and live as a part of Spark itself. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113294#comment-14113294 ] Matei Zaharia commented on SPARK-3215: -- Okay, so my suggestion is do it separately first, and then we can decide whether we commit to offering this when we see the implementation and work out these issues. Add remote interface for SparkContext - Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Marcelo Vanzin Labels: hive Attachments: RemoteSparkContext.pdf A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session. Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system. So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka). I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
