Hi Vivian,
What happens when you run the command in the crontab from the perspective
of the cron user? Are you able to run the cron job under the account of
the ec2-user?
I've used systemd on an ec2 where airflow is run as the user 'airflow'. I
needed to provide an EnvironmentFile in order for the airflow process (and
any process in the conda environment) to work. I'm not sure how a cron
process run as another user picks up similar environment variables.
Contents of airflow-scheduler.service
[Unit]
Description=airflow scheduler
After=multi-user.target
[Service]
User=airflow
Group=airflow
Type=simple
EnvironmentFile=/etc/default/airflow
ExecStart=/home/airflow/anaconda3/envs/py3/bin/airflow scheduler
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
Contents of /etc/default/airflow (well, the template that feeds into this)
# Environment variables to make conda work
PATH={{ anaconda_home_dir }}/envs/py3/bin:{{ ansible_env.PATH }}
CONDA_PATH_BACKUP=/home/airflow/bin:/home/airflow/.local/bin:{{
anaconda_home_dir
}}/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin
CONDA_PREFIX={{ anaconda_home_dir }}/envs/py3
CONDA_DEFAULT_ENV=py3
# Other environment variables
PYTHONPATH={{ airflow_extras_dir }}
AIRFLOW_DAGS_DIR={{ airflow_dags_dir }}
AIRFLOW_S3_BUCKET={{ airflow_s3_bucket }}
Good luck with the issue.
cheers,
Dennis
On Sat, Apr 21, 2018 at 8:09 PM Vivian Wang <vivianw...@transfix.io> wrote:
> Hi,
>
> I use AWS ec2 server:
>
>
> NAME="Amazon Linux AMI"
>
> VERSION="2016.03"
>
> ID="amzn"
>
> ID_LIKE="rhel fedora"
>
> VERSION_ID="2016.03"
>
> PRETTY_NAME="Amazon Linux AMI 2016.03"
>
> ANSI_COLOR="0;33"
>
> CPE_NAME="cpe:/o:amazon:linux:2016.03:ga"
>
> HOME_URL="http://aws.amazon.com/amazon-linux-ami/";
>
> user_id: ec2-user
>
> When I bring up airflow webserver manually using following command, it's
> fine.
>
> /usr/local/bin/airflow webserver -p 8080 -D
>
> but when run this in cronjob:
>
> sudo -u ec2-user -i /usr/local/bin/airflow webserver -p 8080 -D
>
>
> then I got error about this:
> Traceback (most recent call last):
> File “/usr/local/bin/airflow”, line 27, in <module>
> args.func(args)
> File “/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py”, line
> 769, in webserver
> subprocess.Popen(run_args)
> File “/usr/lib64/python2.7/subprocess.py”, line 710, in __init__
> errread, errwrite)
> File “/usr/lib64/python2.7/subprocess.py”, line 1335, in _execute_child
> raise child_exception
> OSError: [Errno 2] No such file or directory
>
> I tried a lot time, but cannot figure out the problem.
> Please help me.
>
> Thanks.
>
> --
>
> Vivian Wang
>
> Sr. Data Engineer
>
> O:
>
> C:
>
> <http://transfix.io>
>
> LinkedIn <https://www.linkedin.com/company/1694643/> | Twitter
> <https://twitter.com/transfixio> | Facebook
> <https://www.facebook.com/transfixIO>
>
> --
> This message may contain confidential and privileged information. If it
> has been sent to you in error, please reply to advise the sender of the
> error and then immediately delete this message.
>
