Hello,

I went to the all-in-one TGZ using maven stuff and it went fine except the job 
is unable to be ran inside mesos...

Of course the path is correct and this command works as expected on my own 
computer... Can you see something weird ? Mesos-slave not really exporting 
SPARK_HOME ? Java issue with semi-colon in path ?


Thanks in advance,


Adam.


+ bash -x spark-dist/spark-1.6.1-bin-hadoop2.6/bin/spark-submit --help
+ '[' -z '' ']'
+++ dirname spark-dist/spark-1.6.1-bin-hadoop2.6/bin/spark-submit
++ cd spark-dist/spark-1.6.1-bin-hadoop2.6/bin/..
++ pwd
+ export 
SPARK_HOME=/tmp/mesos/slaves/9cb693bd-573e-44f3-9b99-a623a4b597ab-S2/frameworks/52c3180b-a1eb-4c79-8c8e-2599d59edcde-0000/executors/ct:1462369865658:0:radar-analysis-via-spark:/runs/4bb42868-fa7b-43a9-a7ca-5ccc8cf92d01/spark-dist/spark-1.6.1-bin-hadoop2.6
+ 
SPARK_HOME=/tmp/mesos/slaves/9cb693bd-573e-44f3-9b99-a623a4b597ab-S2/frameworks/52c3180b-a1eb-4c79-8c8e-2599d59edcde-0000/executors/ct:1462369865658:0:radar-analysis-via-spark:/runs/4bb42868-fa7b-43a9-a7ca-5ccc8cf92d01/spark-dist/spark-1.6.1-bin-hadoop2.6
+ export PYTHONHASHSEED=0
+ PYTHONHASHSEED=0
+ exec 
/tmp/mesos/slaves/9cb693bd-573e-44f3-9b99-a623a4b597ab-S2/frameworks/52c3180b-a1eb-4c79-8c8e-2599d59edcde-0000/executors/ct:1462369865658:0:radar-analysis-via-spark:/runs/4bb42868-fa7b-43a9-a7ca-5ccc8cf92d01/spark-dist/spark-1.6.1-bin-hadoop2.6/bin/spark-class
 org.apache.spark.deploy.SparkSubmit --help
Error: Could not find or load main class org.apache.spark.launcher.Main?


________________________________
De : Cecile, Adam
Envoy? : mercredi 4 mai 2016 09:13
? : user
Objet : RE: Spark job run periodically through Chronos


?That's another option indeed ;)

________________________________
De : Shuai Lin <[email protected]>
Envoy? : mercredi 4 mai 2016 09:12
? : user
Objet : Re: Spark job run periodically through Chronos

Why not using docker? You can build a docker image for your app, and the image 
can be based on https://hub.docker.com/r/mesosphere/spark/ .

On Wed, May 4, 2016 at 2:53 PM, Cecile, Adam 
<[email protected]<mailto:[email protected]>> wrote:

Hello,


I need to run a Spark job every N minutes and I'm wondering what is the 
easier/proper way to do that. I was thinking about using Chronos but if I do 
so, it seems I'll have to use spark-submit anyway.


Should I:

* Bundle spark-submit tool along my app

* Make sure spark-submit command is available on all mesos-slave

* Forget this and do something better


?Thanks in advance,


Best regards, Adam.




Reply via email to