Abhilash Kishore created AIRFLOW-6859:
-----------------------------------------
Summary: Airflow scheduler looking for logs of example DAGs in the
wrong folder
Key: AIRFLOW-6859
URL: https://issues.apache.org/jira/browse/AIRFLOW-6859
Project: Apache Airflow
Issue Type: Bug
Components: scheduler
Affects Versions: 1.10.9
Reporter: Abhilash Kishore
My AIRFLOW_HOME is set to */home/airflow*
I am running Airflow as a service
`cat /etc/systemd/system/airflow-scheduler.service` :
{code:java}
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service
rabbitmq-server.service mssql-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
mssql-server.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
User=airflow
Group=airflow
Type=simple
ExecStart=/usr/local/bin/airflow scheduler
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
{code}
*BUG:*
Airflow scheduler looking for logs of example DAGs in the wrong folder. Example:
{code:java}
Feb 21 01:57:24 airflow-poc airflow[25847]: Process DagFileProcessor281-Process:
Feb 21 01:57:24 airflow-poc airflow[25847]: Traceback (most recent call last):
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
Feb 21 01:57:24 airflow-poc airflow[25847]: self.run()
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
Feb 21 01:57:24 airflow-poc airflow[25847]: self._target(*self._args,
**self._kwargs)
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line
135, in _run_file_processor
Feb 21 01:57:24 airflow-poc airflow[25847]: set_context(log, file_path)
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/local/lib/python3.6/dist-packages/airflow/utils/log/logging_mixin.py",
line 198, in set_context
Feb 21 01:57:24 airflow-poc airflow[25847]: handler.set_context(value)
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/local/lib/python3.6/dist-packages/airflow/utils/log/file_processor_handler.py",
line 66, in set_context
Feb 21 01:57:24 airflow-poc airflow[25847]: self.handler =
logging.FileHandler(local_loc)
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/lib/python3.6/logging/__init__.py", line 1032, in __init__
Feb 21 01:57:24 airflow-poc airflow[25847]: StreamHandler.__init__(self,
self._open())
Feb 21 01:57:24 airflow-poc airflow[25847]: File
"/usr/lib/python3.6/logging/__init__.py", line 1061, in _open
Feb 21 01:57:24 airflow-poc airflow[25847]: return open(self.baseFilename,
self.mode, encoding=self.encoding)
Feb 21 01:57:24 airflow-poc airflow[25847]: PermissionError: [Errno 13]
Permission denied:
'/home/airflow/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log'{code}
The same is happening for all example DAGs (haven't loaded any other DAGs yet).
Airflow scheduler is looking for the logs of the example DAG in
`/home/airflow/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log`.
It should instead look in
`/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log`
instead (without the '/home/airflow' prefix).
In the `dag` table in the database, the DAG with dag_id `example_skip_dag.` has
fileloc
`/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py`.
Not sure why Airflow scheduler is appending `/home/airflow' to this path
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
