[
https://issues.apache.org/jira/browse/AIRFLOW-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ikar Pohorsky updated AIRFLOW-1973:
-----------------------------------
Description:
I'm having a weird (cannot safely reproduce) issue with scheduler log
permissions since version 1.9.0.
*Problem description:*
sometimes (not always) it happens that a new log dir for scheduler is created
with root set as owner (and group) which leads to the scheduler being unable to
write to the logs and therefor no task is scheduled and no task is executed.
Example:
{code}
ubuntu@airflow-admin-master:~$ ls -l airflow/logs/scheduler/
total 24
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 3 13:46 2018-01-03
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 4 07:16 2018-01-04
drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 5 12:08 2018-01-05
drwxr-xr-x 2 root root 4096 Jan 6 00:00 2018-01-06 # fail
drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 7 00:00 2018-01-07
drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 8 00:00 2018-01-08
lrwxrwxrwx 1 ubuntu ubuntu 46 Jan 8 00:00 latest ->
/home/ubuntu/airflow/logs/scheduler/2018-01-08
{code}
the other machine:
{code}
ubuntu@airflow-client-master:~$ ls -l airflow/logs/scheduler/
total 24
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 3 00:00 2018-01-03
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 4 00:00 2018-01-04
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 5 12:25 2018-01-05
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 6 00:00 2018-01-06
drwxr-xr-x 2 root root 4096 Jan 7 00:00 2018-01-07 # fail
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 8 00:00 2018-01-08
lrwxrwxrwx 1 ubuntu ubuntu 46 Jan 8 00:00 latest ->
/home/ubuntu/airflow/logs/scheduler/2018-01-08
{code}
*My setup:*
- _scheduler_ runs on EC2 machine under {{ubuntu:ubuntu}} from systemd service
- _webapp_ runs on the same machine under {{root:root}} in order to be able to
serve on port :80
- _workers_ run on different machines - actually this is irrelevant info;)
Also there are two different "environments" - one for "client" and one for
"admin". The admin one is a fresh installation of airflow 1.9.0. The client one
is an upgrade from 1.8.2 to 1.9.0. Both encounter the permissions problems.
I'm not aware of these problems with 1.8.2 version.
Workaround:
let's get dirty, add following row into root's crontab:
{code}
1 0 * * * chown ubuntu:ubuntu -Rc /home/airflow/logs/scheduler
{code}
was:
I'm having a weird (cannot safely reproduce) issue with log permissions since
version 1.9.0.
*Problem description:*
sometimes (not always) it happens that a new log dir for scheduler is created
with root set as owner (and group) which leads to the scheduler being unable to
write to the logs and therefor no task is scheduled and no task is executed.
Example:
{code}
ubuntu@airflow-admin-master:~$ ls -l airflow/logs/scheduler/
total 24
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 3 13:46 2018-01-03
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 4 07:16 2018-01-04
drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 5 12:08 2018-01-05
drwxr-xr-x 2 root root 4096 Jan 6 00:00 2018-01-06 # fail
drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 7 00:00 2018-01-07
drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 8 00:00 2018-01-08
lrwxrwxrwx 1 ubuntu ubuntu 46 Jan 8 00:00 latest ->
/home/ubuntu/airflow/logs/scheduler/2018-01-08
{code}
the other machine:
{code}
ubuntu@airflow-client-master:~$ ls -l airflow/logs/scheduler/
total 24
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 3 00:00 2018-01-03
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 4 00:00 2018-01-04
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 5 12:25 2018-01-05
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 6 00:00 2018-01-06
drwxr-xr-x 2 root root 4096 Jan 7 00:00 2018-01-07 # fail
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 8 00:00 2018-01-08
lrwxrwxrwx 1 ubuntu ubuntu 46 Jan 8 00:00 latest ->
/home/ubuntu/airflow/logs/scheduler/2018-01-08
{code}
*My setup:*
- _scheduler_ runs on EC2 machine under {{ubuntu:ubuntu}} from systemd service
- _webapp_ runs on the same machine under {{root:root}} in order to be able to
serve on port :80
- _workers_ run on different machines - actually this is irrelevant info;)
Also there are two different "environments" - one for "client" and one for
"admin". The admin one is a fresh installation of airflow 1.9.0. The client one
is an upgrade from 1.8.2 to 1.9.0. Both encounter the permissions problems.
I'm not aware of these problems with 1.8.2 version.
Workaround:
let's get dirty, add following row into root's crontab:
{code}
1 0 * * * chown ubuntu:ubuntu -Rc /home/airflow/logs/scheduler
{code}
> Scheduler log dirs permission issue since airflow 1.9.0
> -------------------------------------------------------
>
> Key: AIRFLOW-1973
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1973
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler, webserver
> Affects Versions: 1.9.0
> Reporter: Ikar Pohorsky
>
> I'm having a weird (cannot safely reproduce) issue with scheduler log
> permissions since version 1.9.0.
> *Problem description:*
> sometimes (not always) it happens that a new log dir for scheduler is created
> with root set as owner (and group) which leads to the scheduler being unable
> to write to the logs and therefor no task is scheduled and no task is
> executed.
> Example:
> {code}
> ubuntu@airflow-admin-master:~$ ls -l airflow/logs/scheduler/
> total 24
> drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 3 13:46 2018-01-03
> drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 4 07:16 2018-01-04
> drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 5 12:08 2018-01-05
> drwxr-xr-x 2 root root 4096 Jan 6 00:00 2018-01-06 # fail
> drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 7 00:00 2018-01-07
> drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 8 00:00 2018-01-08
> lrwxrwxrwx 1 ubuntu ubuntu 46 Jan 8 00:00 latest ->
> /home/ubuntu/airflow/logs/scheduler/2018-01-08
> {code}
> the other machine:
> {code}
> ubuntu@airflow-client-master:~$ ls -l airflow/logs/scheduler/
> total 24
> drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 3 00:00 2018-01-03
> drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 4 00:00 2018-01-04
> drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 5 12:25 2018-01-05
> drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 6 00:00 2018-01-06
> drwxr-xr-x 2 root root 4096 Jan 7 00:00 2018-01-07 # fail
> drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 8 00:00 2018-01-08
> lrwxrwxrwx 1 ubuntu ubuntu 46 Jan 8 00:00 latest ->
> /home/ubuntu/airflow/logs/scheduler/2018-01-08
> {code}
> *My setup:*
> - _scheduler_ runs on EC2 machine under {{ubuntu:ubuntu}} from systemd service
> - _webapp_ runs on the same machine under {{root:root}} in order to be able
> to serve on port :80
> - _workers_ run on different machines - actually this is irrelevant info;)
> Also there are two different "environments" - one for "client" and one for
> "admin". The admin one is a fresh installation of airflow 1.9.0. The client
> one is an upgrade from 1.8.2 to 1.9.0. Both encounter the permissions
> problems.
> I'm not aware of these problems with 1.8.2 version.
> Workaround:
> let's get dirty, add following row into root's crontab:
> {code}
> 1 0 * * * chown ubuntu:ubuntu -Rc /home/airflow/logs/scheduler
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
