Hello,
In the meanwhile I figured out what's wrong: Spark fails to handle SPARK_HOME if the path contains colon characters... I'll open a bug ! Regards, Adam. ________________________________ De : haosdent <[email protected]> Envoyé : dimanche 8 mai 2016 19:03 À : user Objet : Re: Spark job run periodically through Chronos The error looks like your classpath for spark is incorrect. I think this is a spark usage problem. You could take a look at https://github.com/apache/spark/blob/master/bin/spark-class to check if the jars exist under "${SPARK_HOME}/jars". Or get more information about this via send another email to [email protected]<mailto:[email protected]>. On Wed, May 4, 2016 at 9:55 PM, Cecile, Adam <[email protected]<mailto:[email protected]>> wrote: Hello, I went to the all-in-one TGZ using maven stuff and it went fine except the job is unable to be ran inside mesos... Of course the path is correct and this command works as expected on my own computer... Can you see something weird ? Mesos-slave not really exporting SPARK_HOME ? Java issue with semi-colon in path ? Thanks in advance, Adam. + bash -x spark-dist/spark-1.6.1-bin-hadoop2.6/bin/spark-submit --help + '[' -z '' ']' +++ dirname spark-dist/spark-1.6.1-bin-hadoop2.6/bin/spark-submit ++ cd spark-dist/spark-1.6.1-bin-hadoop2.6/bin/.. ++ pwd + export SPARK_HOME=/tmp/mesos/slaves/9cb693bd-573e-44f3-9b99-a623a4b597ab-S2/frameworks/52c3180b-a1eb-4c79-8c8e-2599d59edcde-0000/executors/ct:1462369865658:0:radar-analysis-via-spark:/runs/4bb42868-fa7b-43a9-a7ca-5ccc8cf92d01/spark-dist/spark-1.6.1-bin-hadoop2.6 + SPARK_HOME=/tmp/mesos/slaves/9cb693bd-573e-44f3-9b99-a623a4b597ab-S2/frameworks/52c3180b-a1eb-4c79-8c8e-2599d59edcde-0000/executors/ct:1462369865658:0:radar-analysis-via-spark:/runs/4bb42868-fa7b-43a9-a7ca-5ccc8cf92d01/spark-dist/spark-1.6.1-bin-hadoop2.6 + export PYTHONHASHSEED=0 + PYTHONHASHSEED=0 + exec /tmp/mesos/slaves/9cb693bd-573e-44f3-9b99-a623a4b597ab-S2/frameworks/52c3180b-a1eb-4c79-8c8e-2599d59edcde-0000/executors/ct:1462369865658:0:radar-analysis-via-spark:/runs/4bb42868-fa7b-43a9-a7ca-5ccc8cf92d01/spark-dist/spark-1.6.1-bin-hadoop2.6/bin/spark-class org.apache.spark.deploy.SparkSubmit --help Error: Could not find or load main class org.apache.spark.launcher.Main ________________________________ De : Cecile, Adam Envoyé : mercredi 4 mai 2016 09:13 À : user Objet : RE: Spark job run periodically through Chronos That's another option indeed ;) ________________________________ De : Shuai Lin <[email protected]<mailto:[email protected]>> Envoyé : mercredi 4 mai 2016 09:12 À : user Objet : Re: Spark job run periodically through Chronos Why not using docker? You can build a docker image for your app, and the image can be based on https://hub.docker.com/r/mesosphere/spark/ . On Wed, May 4, 2016 at 2:53 PM, Cecile, Adam <[email protected]<mailto:[email protected]>> wrote: Hello, I need to run a Spark job every N minutes and I'm wondering what is the easier/proper way to do that. I was thinking about using Chronos but if I do so, it seems I'll have to use spark-submit anyway. Should I: * Bundle spark-submit tool along my app * Make sure spark-submit command is available on all mesos-slave * Forget this and do something better Thanks in advance, Best regards, Adam. -- Best Regards, Haosdent Huang

