​Hello,

In the meanwhile I figured out what's wrong: Spark fails to handle SPARK_HOME 
if the path contains colon characters... I'll open a bug !


Regards, Adam.

________________________________
De : haosdent <[email protected]>
Envoyé : dimanche 8 mai 2016 19:03
À : user
Objet : Re: Spark job run periodically through Chronos

The error looks like your classpath for spark is incorrect. I think this is a 
spark usage problem.
You could take a look at 
https://github.com/apache/spark/blob/master/bin/spark-class to check
if the jars exist under "${SPARK_HOME}/jars". Or get more information about 
this via send another email to 
[email protected]<mailto:[email protected]>.

On Wed, May 4, 2016 at 9:55 PM, Cecile, Adam 
<[email protected]<mailto:[email protected]>> wrote:

Hello,


I went to the all-in-one TGZ using maven stuff and it went fine except the job 
is unable to be ran inside mesos...

Of course the path is correct and this command works as expected on my own 
computer... Can you see something weird ? Mesos-slave not really exporting 
SPARK_HOME ? Java issue with semi-colon in path ?


Thanks in advance,


Adam.


+ bash -x spark-dist/spark-1.6.1-bin-hadoop2.6/bin/spark-submit --help
+ '[' -z '' ']'
+++ dirname spark-dist/spark-1.6.1-bin-hadoop2.6/bin/spark-submit
++ cd spark-dist/spark-1.6.1-bin-hadoop2.6/bin/..
++ pwd
+ export 
SPARK_HOME=/tmp/mesos/slaves/9cb693bd-573e-44f3-9b99-a623a4b597ab-S2/frameworks/52c3180b-a1eb-4c79-8c8e-2599d59edcde-0000/executors/ct:1462369865658:0:radar-analysis-via-spark:/runs/4bb42868-fa7b-43a9-a7ca-5ccc8cf92d01/spark-dist/spark-1.6.1-bin-hadoop2.6
+ 
SPARK_HOME=/tmp/mesos/slaves/9cb693bd-573e-44f3-9b99-a623a4b597ab-S2/frameworks/52c3180b-a1eb-4c79-8c8e-2599d59edcde-0000/executors/ct:1462369865658:0:radar-analysis-via-spark:/runs/4bb42868-fa7b-43a9-a7ca-5ccc8cf92d01/spark-dist/spark-1.6.1-bin-hadoop2.6
+ export PYTHONHASHSEED=0
+ PYTHONHASHSEED=0
+ exec 
/tmp/mesos/slaves/9cb693bd-573e-44f3-9b99-a623a4b597ab-S2/frameworks/52c3180b-a1eb-4c79-8c8e-2599d59edcde-0000/executors/ct:1462369865658:0:radar-analysis-via-spark:/runs/4bb42868-fa7b-43a9-a7ca-5ccc8cf92d01/spark-dist/spark-1.6.1-bin-hadoop2.6/bin/spark-class
 org.apache.spark.deploy.SparkSubmit --help
Error: Could not find or load main class org.apache.spark.launcher.Main​


________________________________
De : Cecile, Adam
Envoyé : mercredi 4 mai 2016 09:13
À : user
Objet : RE: Spark job run periodically through Chronos


​That's another option indeed ;)

________________________________
De : Shuai Lin <[email protected]<mailto:[email protected]>>
Envoyé : mercredi 4 mai 2016 09:12
À : user
Objet : Re: Spark job run periodically through Chronos

Why not using docker? You can build a docker image for your app, and the image 
can be based on https://hub.docker.com/r/mesosphere/spark/ .

On Wed, May 4, 2016 at 2:53 PM, Cecile, Adam 
<[email protected]<mailto:[email protected]>> wrote:

Hello,


I need to run a Spark job every N minutes and I'm wondering what is the 
easier/proper way to do that. I was thinking about using Chronos but if I do 
so, it seems I'll have to use spark-submit anyway.


Should I:

* Bundle spark-submit tool along my app

* Make sure spark-submit command is available on all mesos-slave

* Forget this and do something better


​Thanks in advance,


Best regards, Adam.







--
Best Regards,
Haosdent Huang

Reply via email to