[ 
https://issues.apache.org/jira/browse/AIRFLOW-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor updated AIRFLOW-2729:
---------------------------------------
    Description: 
I have a repo that in 1.10 is giving airflowignore errors that did not exist in 
1.9. I have a DAG repo with the following .airflowignore:

{{airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/}}
{{total 172}}
{{drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .}}
{{drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..}}
{{-rw-r--r-- 1 airflow airflow 13 Jul 9 16:20 .airflowignore}}

{{airflow@airflow-core-i-063df3268720e58fd:~$ cat 
/home/airflow/airflow/dags/airflow-tasks/.airflowignore}}
{{submodules/*}}

However, the submoduled repository is being scanned for DAGs anyways, including 
the test suite. Note the paths in the section below:

 

{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,814] 
\{{models.py:351}} DEBUG - Importing 
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/quantflow/operators/zipline_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,817] 
\{{models.py:351}} DEBUG - Importing 
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,818] 
\{{models.py:365}} ERROR - Failed to import: 
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
 line 362, in process_file}}
{{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name, 
filepath)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in 
load_source}}
{{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 693, in _load}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 673, in _load_unlocked}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
importlib._bootstrap_external>", line 665, in exec_module}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 222, in _call_with_frames_removed}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py",
 line 6, in <module>}}
{{Jul 09 18:52:01 airflow_web-stdout.log: from moto import mock_sqs}}
{{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named 'moto'}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,821] 
\{{models.py:351}} DEBUG - Importing 
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,822] 
\{{models.py:365}} ERROR - Failed to import: 
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
 line 362, in process_file}}
{{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name, 
filepath)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in 
load_source}}
{{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 693, in _load}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 673, in _load_unlocked}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
importlib._bootstrap_external>", line 665, in exec_module}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 222, in _call_with_frames_removed}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py",
 line 6, in <module>}}
{{Jul 09 18:52:01 airflow_web-stdout.log: from freezegun import freeze_time}}
{{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named 
'freezegun'}}

 

 

  was:
I have a repo that in 1.10 is giving airflowignore errors that did not exist in 
1.9. I have a DAG repo with the following .airflowignore:

{{airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/}}
{{total 172}}
{{drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .}}
{{drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..}}
{{-rw-r--r-- 1 airflow airflow 13 Jul 9 16:20 .airflowignore}}

{{airflow@airflow-core-i-063df3268720e58fd:~$ cat 
/home/airflow/airflow/dags/airflow-tasks/.airflowignore }}
{{submodules/*}}

However, the submoduled repository is being scanned for DAGs anyways, including 
the test suite. Note the paths in the section below:

 

{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,814] 
\{{models.py:351}} DEBUG - Importing 
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/quantflow/operators/zipline_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,817] 
\{{models.py:351}} DEBUG - Importing 
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,818] 
\{{models.py:365}} ERROR - Failed to import: 
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
 line 362, in process_file}}
{{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name, 
filepath)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in 
load_source}}
{{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 693, in _load}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 673, in _load_unlocked}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
importlib._bootstrap_external>", line 665, in exec_module}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 222, in _call_with_frames_removed}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py",
 line 6, in <module>}}
{{Jul 09 18:52:01 airflow_web-stdout.log: from moto import mock_sqs}}
{{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named 'moto'}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,821] 
\{{models.py:351}} DEBUG - Importing 
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,822] 
\{{models.py:365}} ERROR - Failed to import: 
/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
{{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
 line 362, in process_file}}
{{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name, 
filepath)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in 
load_source}}
{{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 693, in _load}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 673, in _load_unlocked}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
importlib._bootstrap_external>", line 665, in exec_module}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", 
line 222, in _call_with_frames_removed}}
{{Jul 09 18:52:01 airflow_web-stdout.log: File 
"/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py",
 line 6, in <module>}}
{{Jul 09 18:52:01 airflow_web-stdout.log: from freezegun import freeze_time}}
{{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named 
'freezegun'}}

 

 


> .airflowignore is not being respected
> -------------------------------------
>
>                 Key: AIRFLOW-2729
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2729
>             Project: Apache Airflow
>          Issue Type: Bug
>    Affects Versions: 1.10
>            Reporter: James Meickle
>            Priority: Minor
>
> I have a repo that in 1.10 is giving airflowignore errors that did not exist 
> in 1.9. I have a DAG repo with the following .airflowignore:
> {{airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/}}
> {{total 172}}
> {{drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .}}
> {{drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..}}
> {{-rw-r--r-- 1 airflow airflow 13 Jul 9 16:20 .airflowignore}}
> {{airflow@airflow-core-i-063df3268720e58fd:~$ cat 
> /home/airflow/airflow/dags/airflow-tasks/.airflowignore}}
> {{submodules/*}}
> However, the submoduled repository is being scanned for DAGs anyways, 
> including the test suite. Note the paths in the section below:
>  
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,814] 
> \{{models.py:351}} DEBUG - Importing 
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/quantflow/operators/zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,817] 
> \{{models.py:351}} DEBUG - Importing 
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,818] 
> \{{models.py:365}} ERROR - Failed to import: 
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File 
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
>  line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name, 
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File 
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in 
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
> importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
> importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
> importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
> importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File 
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py",
>  line 6, in <module>}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: from moto import mock_sqs}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named 
> 'moto'}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,821] 
> \{{models.py:351}} DEBUG - Importing 
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,822] 
> \{{models.py:365}} ERROR - Failed to import: 
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File 
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
>  line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name, 
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File 
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in 
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
> importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
> importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
> importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen 
> importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File 
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py",
>  line 6, in <module>}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: from freezegun import freeze_time}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named 
> 'freezegun'}}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to