Re: Problem mixing MESOS Cluster Mode and Docker task execution

[yanlin wang]

How you guys make driver docker within container to be reachable from spark 
worker ? 

Would you share your driver docker? i am trying to put only driver in docker 
and spark running with yarn outside of container and i don’t want to use 
—net=host 

Thx
Yanlin

> On Mar 10, 2016, at 11:06 AM, Guillaume Eynard Bontemps 
>  wrote:
> 
> Glad to hear it. Thanks all  for sharing your  solutions.
> 
> 
> Le jeu. 10 mars 2016 19:19, Eran Chinthaka Withana  > a écrit :
> Phew, it worked. All I had to do was to add export 
> SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6"
>  before calling spark-submit. Guillaume, thanks for the pointer. 
> 
> Timothy, thanks for looking into this. Looking forward to see a fix soon. 
> 
> Thanks,
> Eran Chinthaka Withana
> 
> On Thu, Mar 10, 2016 at 10:10 AM, Tim Chen  > wrote:
> Hi Eran,
> 
> I need to investigate but perhaps that's true, we're using SPARK_JAVA_OPTS to 
> pass all the options and not --conf.
> 
> I'll take a look at the bug, but if you can try the workaround and see if 
> that fixes your problem.
> 
> Tim
> 
> On Thu, Mar 10, 2016 at 10:08 AM, Eran Chinthaka Withana 
> mailto:eran.chinth...@gmail.com>> wrote:
> Hi Timothy
> 
> What version of spark are you guys running?
> 
> I'm using Spark 1.6.0. You can see the Dockerfile I used here: 
> https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile
>  
> 
>  
>  
> And also did you set the working dir in your image to be spark home?
> 
> Yes I did. You can see it here: https://goo.gl/8PxtV8 
> 
> Can it be because of this: https://issues.apache.org/jira/browse/SPARK-13258 
>  as Guillaume pointed out 
> above? As you can see, I'm passing in the docker image URI through 
> spark-submit (--conf 
> spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6)
> 
> Thanks,
> Eran
> 
> 
> 
> 

spark driver in docker

[yanlin wang]

We would like to run multiple spark driver in docker container. Any suggestion 
for the port expose and network settings for docker so driver is reachable by 
the worker nodes? —net=“hosts” is the last thing we want to do.

Thx
Yanlin
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Building a REST Service with Spark back-end

[yanlin wang]

Did any one use Livy in real world high concurrency web app? I think it uses 
spark submit command line to create job... How about  job server or notebook 
comparing with Livy?

Thx,
Yanlin

Sent from my iPhone

> On Mar 2, 2016, at 6:24 AM, Guru Medasani  wrote:
> 
> Hi Don,
> 
> Here is another REST interface for interacting with Spark from anywhere. 
> 
> https://github.com/cloudera/livy
> 
> Here is an example to estimate PI using Spark from Python using requests 
> library. 
> 
> >>> data = {
> ...   'code': textwrap.dedent("""\
> ...  val NUM_SAMPLES = 10;
> ...  val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
> ...val x = Math.random();
> ...val y = Math.random();
> ...if (x*x + y*y < 1) 1 else 0
> ...  }.reduce(_ + _);
> ...  println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
> ...  """)
> ... }
> >>> r = requests.post(statements_url, data=json.dumps(data), headers=headers)
> >>> pprint.pprint(r.json())
> {u'id': 1,
>  u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: 
> Int = 10\ncount: Int = 78501'},
>  u'execution_count': 1,
>  u'status': u'ok'},
>  u'state': u'available'}
> 
> 
> Guru Medasani
> gdm...@gmail.com
> 
> 
> 
>> On Mar 2, 2016, at 7:47 AM, Todd Nist  wrote:
>> 
>> Have you looked at Apache Toree, http://toree.apache.org/.  This was 
>> formerly the Spark-Kernel from IBM but contributed to apache.
>> 
>> https://github.com/apache/incubator-toree
>> 
>> You can find a good overview on the spark-kernel here:
>> http://www.spark.tc/how-to-enable-interactive-applications-against-apache-spark/
>> 
>> Not sure if that is of value to you or not.
>> 
>> HTH.
>> 
>> -Todd
>> 
>>> On Tue, Mar 1, 2016 at 7:30 PM, Don Drake  wrote:
>>> I'm interested in building a REST service that utilizes a Spark SQL Context 
>>> to return records from a DataFrame (or IndexedRDD?) and even add/update 
>>> records.
>>> 
>>> This will be a simple REST API, with only a few end-points.  I found this 
>>> example:
>>> 
>>> https://github.com/alexmasselot/spark-play-activator
>>> 
>>> which looks close to what I am interested in doing.  
>>> 
>>> Are there any other ideas or options if I want to run this in a YARN 
>>> cluster?
>>> 
>>> Thanks.
>>> 
>>> -Don
>>> 
>>> -- 
>>> Donald Drake
>>> Drake Consulting
>>> http://www.drakeconsulting.com/
>>> https://twitter.com/dondrake
>>> 800-733-2143
>