Hi John, I'm currently working on a cluster mode design a PoC, but it is also not sharing drivers as Spark AFAIK is designed to not share drivers between apps.
The cluster mode for Mesos is going to be a way to submit apps to your cluster, and each app will be running in the cluster as a new driver that is managed by a cluster dispatcher, and you don't need to wait for the client to finish to get all the results. I'll be updating the JIRA and PR once I have this ready, which is aimed for this next release. Tim On Fri, Feb 20, 2015 at 8:09 AM, John Omernik <[email protected]> wrote: > Tim - on the Spark list your name was brought up in relation to > https://issues.apache.org/jira/browse/SPARK-5338 I asked this question > there but I'll ask it here too, what can I do to help on this. I am > not a coder unfortunately, but I am user willing to try things :) This > looks really cool for what we would like to do with Spark and Mesos > and I'd love to be able to contribute and/or get an understanding of a > (even tentative) timeline. I am not trying to be pushy, I understand > lots of things are likely on your agenda :) > > John > > > > On Tue, Feb 17, 2015 at 6:33 AM, John Omernik <[email protected]> wrote: > > Tim, thanks, that makes sense, the checking for ports and incrementing > > was new to me, so hearing about that helps. Next question.... is it > > possible, for a driver to be shared by the same user some how? This > > would be desirable from the standpoint of running an iPython notebook > > server (Jupyter Hub). I have it setup that every time a notebook is > > opened, that the imports for spark are run, (the idea is the > > environment is ready to go for analysis) however, if each user, has 5 > > notebooks open at any time, that would be a lot of spark drivers! But, > > I suppose before asking that, I should ask about the sequence of > > drivers... are they serial? i.e. can one driver server only one query > > at a time? What is the optimal size for a driver (in memory) what > > does the memory affect in the driver? I.e. is a driver with smaller > > amounts of memory limited in the number of results etc? > > > > Lots of questions here, if these are more spark related questions, let > > me know, I can hop over to spark users, but since I am curious on > > spark on mesos, I figured I'd try here first. > > > > Thanks for your help! > > > > > > > > On Mon, Feb 16, 2015 at 10:30 AM, Tim Chen <[email protected]> wrote: > >> Hi John, > >> > >> With Spark on Mesos, each client (spark-submit) starts a SparkContext > which > >> initializes its own SparkUI and framework. There is a default 4040 for > the > >> Spark UI port, but if it's occupied Spark automatically tries ports > >> incrementally for you, so your next could be 4041 if it's available. > >> > >> Driver is not shared between user, each user creates its own driver. > >> > >> About slowness it's hard to say without any information, you need to > tell us > >> your cluster setup, what mode you're Mesos with and if there is anything > >> else running in the cluster, the job, etc. > >> > >> Tim > >> > >> On Sat, Feb 14, 2015 at 5:06 PM, John Omernik <[email protected]> wrote: > >>> > >>> Hello all, I am running Spark on Mesos and I think I am love, but I > >>> have some questions. I am running the python shell via iPython > >>> Notebooks (Jupyter) and it works great, but I am trying to figure out > >>> how things are actually submitted... like for example, when I submit > >>> the spark app from the iPython notebook server, I am opening a new > >>> kernel and I see a new spark submit (similar to the below) for each > >>> kernel... but, how is that actually working on the cluster, I can > >>> connect to the spark server UI on 4040, but shouldn't there be a > >>> different one for each driver? Is that causing conflicts? after a > >>> while things seem to run slow is this due to some weird conflicts? > >>> Should I be specifying unique ports for each server? Is the driver > >>> shared between users? what about between kerne's for the same user? > >>> Curious if anyone has any insight. > >>> > >>> Thanks! > >>> > >>> > >>> java org.apache.spark.deploy.SparkSubmitDriverBootstrapper --master > >>> mesos://hadoopmapr3:5050 --driver-memory 1G --executor-memory 4096M > >>> pyspark-shell > >> > >> >
