HI , Please check your deploy mode and master , For example if you want to deploy in yarn cluster you should use --master yarn-cluster , if you want to do it on yarn client mode you should use --master yarn-client .
Please note for your case deploying yarn-cluster will be better as cluster mode is async , it will not die if the launcher dies, in yarn-client mode your driver die as soon as launcher die so as your executors . Regards, Rabin On Wed, Jul 20, 2016 at 3:41 PM, focus <focushe...@qq.com> wrote: > Hi, I just meet this problem, too! The reason is crontab runtime doesn't > have the variables you defined, such as $SPARK_HOME. > I defined the $SPARK_HOME and other variables in /etc/profile like this: > > export $MYSCRIPTS=/opt/myscripts > export $SPARK_HOME=/opt/spark > > then, in my crontab job script daily_job.sh > > #!/bin/sh > > . /etc/profile > > $SPARK_HOME/bin/spark-submit $MYSCRIPTS/fix_fh_yesterday.py > > then, in crontab -e > > 0 8 * * * /home/user/daily_job.sh > > hope this helps~ > > > > > ------------------ Original ------------------ > *From:* "luohui20001"<luohui20...@sina.com>; > *Date:* 2016年7月20日(星期三) 晚上6:00 > *To:* "user@spark.apache.org"<user@spark.apache.org>; > *Subject:* run spark apps in linux crontab > > hi guys: > I add a spark-submit job into my Linux crontab list by the means > below ,however none of them works. If I change it to a normal shell script, > it is ok. I don't quite understand why. I checked the 8080 web ui of my > spark cluster, no job submitted, and there is not messages in > /home/hadoop/log. > Any idea is welcome. > > [hadoop@master ~]$ crontab -e > 1. > 22 21 * * * sh /home/hadoop/shellscripts/run4.sh > /home/hadoop/log > > and in run4.sh,it wrote: > $SPARK_HOME/bin/spark-submit --class com.abc.myclass > --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar > $SPARK_HOME/MyJar.jar > /home/hadoop/log > > 2. > 22 21 * * * $SPARK_HOME/bin/spark-submit --class com.abc.myclass > --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar > $SPARK_HOME/MyJar.jar > /home/hadoop/log > > 3. > 22 21 * * * /usr/lib/spark/bin/spark-submit --class com.abc.myclass > --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar > /usr/lib/spark/MyJar.jar > /home/hadoop/log > > 4. > 22 21 * * * hadoop /usr/lib/spark/bin/spark-submit --class com.abc.myclass > --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar > /usr/lib/spark/MyJar.jar > /home/hadoop/log > > -------------------------------- > > Thanks&Best regards! > San.Luo >
