[jira] [Updated] (AIRFLOW-2729) .airflowignore is not being respected
[
https://issues.apache.org/jira/browse/AIRFLOW-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin updated AIRFLOW-2729:
Fix Version/s: 1.10.0
> .airflowignore is not being respected
> -
>
> Key: AIRFLOW-2729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2729
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10
>Reporter: James Meickle
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 1.10.0
>
>
> I have a repo that in 1.10 is giving airflowignore errors that did not exist
> in 1.9. I have a DAG repo with the following .airflowignore:
> {{airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/}}
> {{total 172}}
> {{drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .}}
> {{drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..}}
> {{-rw-r--r-- 1 airflow airflow 13 Jul 9 16:20 .airflowignore}}
> {{airflow@airflow-core-i-063df3268720e58fd:~$ cat
> /home/airflow/airflow/dags/airflow-tasks/.airflowignore}}
> {{submodules/*}}
> However, the submoduled repository is being scanned for DAGs anyways,
> including the test suite. Note the paths in the section below:
>
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,814]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/quantflow/operators/zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,817]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,818]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py",
> line 6, in }}
> {{Jul 09 18:52:01 airflow_web-stdout.log: from moto import mock_sqs}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named
> 'moto'}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,821]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,822]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py",
> line 6, in }}
> {{Jul 09 18:52:01 airflow_web-stdout.log: from freezegun import freeze_time}}
> {{Jul 09 18:52:01 airflow_web-std
[jira] [Updated] (AIRFLOW-2729) .airflowignore is not being respected
[
https://issues.apache.org/jira/browse/AIRFLOW-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-2729:
---
Description:
I have a repo that in 1.10 is giving airflowignore errors that did not exist in
1.9. I have a DAG repo with the following .airflowignore:
{{airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/}}
{{total 172}}
{{drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .}}
{{drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..}}
{{-rw-r--r-- 1 airflow airflow 13 Jul 9 16:20 .airflowignore}}
{{airflow@airflow-core-i-063df3268720e58fd:~$ cat
/home/airflow/airflow/dags/airflow-tasks/.airflowignore}}
{{submodules/*}}
However, the submoduled repository is being scanned for DAGs anyways, including
the test suite. Note the paths in the section below:
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,814]
\{{models.py:351}} DEBUG - Importing
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/quantflow/operators/zipline_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,817]
\{{models.py:351}} DEBUG - Importing
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,818]
\{{models.py:365}} ERROR - Failed to import:
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File
"/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
line 362, in process_file}}
{{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
filepath)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File
"/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
load_source}}
{{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "",
line 693, in _load}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "",
line 673, in _load_unlocked}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "", line 665, in exec_module}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "",
line 222, in _call_with_frames_removed}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File
"/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py",
line 6, in }}
{{Jul 09 18:52:01 airflow_web-stdout.log: from moto import mock_sqs}}
{{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named 'moto'}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,821]
\{{models.py:351}} DEBUG - Importing
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,822]
\{{models.py:365}} ERROR - Failed to import:
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File
"/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
line 362, in process_file}}
{{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
filepath)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File
"/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
load_source}}
{{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "",
line 693, in _load}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "",
line 673, in _load_unlocked}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "", line 665, in exec_module}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "",
line 222, in _call_with_frames_removed}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File
"/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py",
line 6, in }}
{{Jul 09 18:52:01 airflow_web-stdout.log: from freezegun import freeze_time}}
{{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named
'freezegun'}}
was:
I have a repo that in 1.10 is giving airflowignore errors that did not exist in
1.9. I have a DAG repo with the following .airflowignore:
{{airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/}}
{{total 172}}
{{drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .}}
{{drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..}}
{{-rw-r--r-- 1 airflow airflow 13 Jul 9 16:20 .airflowignore}}
{{airflow@airflow-core-i-063df3268720e58fd:~$ cat
/home/airflow/airflow/dags/airflow-tasks/.airflowignore }}
{{submodules/*}}
However, the submoduled repository is being scanned for DAGs anyways, including
the test suite. Note the paths in the section below:
{{Jul 09 18:52:01 airfl
