got it. difference:> : all messages goes to the log file, leaving no messages in STDOUTtee: all message goes to the log file and STDOUT at the same time.
-------------------------------- Thanks&Best regards! San.Luo ----- 原始邮件 ----- 发件人：Chanh Le <[email protected]> 收件人：[email protected] 抄送人：focus <[email protected]>, user <[email protected]> 主题：Re: run spark apps in linux crontab 日期：2016年07月21日 11点38分 you should you use command.sh | tee file.log On Jul 21, 2016, at 10:36 AM, <[email protected]> <[email protected]> wrote: thank you focus, and all.this problem solved by adding a line ". /etc/profile" in my shell. -------------------------------- Thanks&Best regards! San.Luo ----- 原始邮件 ----- 发件人："focus" <[email protected]> 收件人："luohui20001" <[email protected]>, "[email protected]" <[email protected]> 主题：Re:run spark apps in linux crontab 日期：2016年07月20日 18点11分 Hi, I just meet this problem, too! The reason is crontab runtime doesn't have the variables you defined, such as $SPARK_HOME.I defined the $SPARK_HOME and other variables in /etc/profile like this: export $MYSCRIPTS=/opt/myscriptsexport $SPARK_HOME=/opt/spark then, in my crontab job script daily_job.sh #!/bin/sh . /etc/profile $SPARK_HOME/bin/spark-submit $MYSCRIPTS/fix_fh_yesterday.py then, in crontab -e 0 8 * * * /home/user/daily_job.sh hope this helps~ ------------------ Original ------------------From: "luohui20001"<[email protected]>; Date: 2016年7月20日(星期三) 晚上6:00To: "[email protected]"<[email protected]>; Subject: run spark apps in linux crontab hi guys: I add a spark-submit job into my Linux crontab list by the means below ,however none of them works. If I change it to a normal shell script, it is ok. I don't quite understand why. I checked the 8080 web ui of my spark cluster, no job submitted, and there is not messages in /home/hadoop/log. Any idea is welcome. [hadoop@master ~]$ crontab -e1.22 21 * * * sh /home/hadoop/shellscripts/run4.sh > /home/hadoop/log and in run4.sh,it wrote:$SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log 2. 22 21 * * * $SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log 3.22 21 * * * /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log 4.22 21 * * * hadoop /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log -------------------------------- Thanks&Best regards! San.Luo
