[jira] [Commented] (AIRFLOW-1238) Connections CLI doesn't decode percent encodings
[
https://issues.apache.org/jira/browse/AIRFLOW-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16543881#comment-16543881
]
Matt Chapman commented on AIRFLOW-1238:
---
https://github.com/apache/incubator-airflow/pull/3605
> Connections CLI doesn't decode percent encodings
>
>
> Key: AIRFLOW-1238
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1238
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: John Zeringue
>Assignee: Matt Chapman
>Priority: Minor
>
> For example, {{airflow connections -a --conn_id my_connection --conn_uri
> https://my%20username:my%20passw...@mydomain.com}} should create a connection
> with username {{my username}} and password {{my password}}. Instead, it
> creates a connection with username {{my%20username}} and password
> {{my%20password}}.
> As it stands now, email usernames (which have an {{@}}) or passwords with
> special characters like {{p@$$word}} can't be specified from the CLI either.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1238) Connections CLI doesn't decode percent encodings
[
https://issues.apache.org/jira/browse/AIRFLOW-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Chapman reassigned AIRFLOW-1238:
-
Assignee: Matt Chapman
> Connections CLI doesn't decode percent encodings
>
>
> Key: AIRFLOW-1238
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1238
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: John Zeringue
>Assignee: Matt Chapman
>Priority: Minor
>
> For example, {{airflow connections -a --conn_id my_connection --conn_uri
> https://my%20username:my%20passw...@mydomain.com}} should create a connection
> with username {{my username}} and password {{my password}}. Instead, it
> creates a connection with username {{my%20username}} and password
> {{my%20password}}.
> As it stands now, email usernames (which have an {{@}}) or passwords with
> special characters like {{p@$$word}} can't be specified from the CLI either.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2752) Use logging instead of stdout
Fokko Driesprong created AIRFLOW-2752: - Summary: Use logging instead of stdout Key: AIRFLOW-2752 URL: https://issues.apache.org/jira/browse/AIRFLOW-2752 Project: Apache Airflow Issue Type: Bug Reporter: Fokko Driesprong -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2729) .airflowignore is not being respected
[
https://issues.apache.org/jira/browse/AIRFLOW-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin closed AIRFLOW-2729.
---
Resolution: Fixed
> .airflowignore is not being respected
> -
>
> Key: AIRFLOW-2729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2729
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10
>Reporter: James Meickle
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 1.10.0
>
>
> I have a repo that in 1.10 is giving airflowignore errors that did not exist
> in 1.9. I have a DAG repo with the following .airflowignore:
> {{airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/}}
> {{total 172}}
> {{drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .}}
> {{drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..}}
> {{-rw-r--r-- 1 airflow airflow 13 Jul 9 16:20 .airflowignore}}
> {{airflow@airflow-core-i-063df3268720e58fd:~$ cat
> /home/airflow/airflow/dags/airflow-tasks/.airflowignore}}
> {{submodules/*}}
> However, the submoduled repository is being scanned for DAGs anyways,
> including the test suite. Note the paths in the section below:
>
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,814]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/quantflow/operators/zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,817]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,818]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py",
> line 6, in }}
> {{Jul 09 18:52:01 airflow_web-stdout.log: from moto import mock_sqs}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named
> 'moto'}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,821]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,822]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py",
> line 6, in }}
> {{Jul 09 18:52:01 airflow_web-stdout.log: from freezegun import freeze_time}}
> {{Jul 09 18:52:01 airflow_web-stdout.lo
[jira] [Updated] (AIRFLOW-2729) .airflowignore is not being respected
[
https://issues.apache.org/jira/browse/AIRFLOW-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin updated AIRFLOW-2729:
Fix Version/s: 1.10.0
> .airflowignore is not being respected
> -
>
> Key: AIRFLOW-2729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2729
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10
>Reporter: James Meickle
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 1.10.0
>
>
> I have a repo that in 1.10 is giving airflowignore errors that did not exist
> in 1.9. I have a DAG repo with the following .airflowignore:
> {{airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/}}
> {{total 172}}
> {{drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .}}
> {{drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..}}
> {{-rw-r--r-- 1 airflow airflow 13 Jul 9 16:20 .airflowignore}}
> {{airflow@airflow-core-i-063df3268720e58fd:~$ cat
> /home/airflow/airflow/dags/airflow-tasks/.airflowignore}}
> {{submodules/*}}
> However, the submoduled repository is being scanned for DAGs anyways,
> including the test suite. Note the paths in the section below:
>
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,814]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/quantflow/operators/zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,817]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,818]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py",
> line 6, in }}
> {{Jul 09 18:52:01 airflow_web-stdout.log: from moto import mock_sqs}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named
> 'moto'}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,821]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,822]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py",
> line 6, in }}
> {{Jul 09 18:52:01 airflow_web-stdout.log: from freezegun import freeze_time}}
> {{Jul 09 18:52:01 airflow_web-std
[jira] [Updated] (AIRFLOW-1729) Ignore whole directories in .airflowignore
[
https://issues.apache.org/jira/browse/AIRFLOW-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin updated AIRFLOW-1729:
Fix Version/s: (was: 2.0.0)
1.10.0
> Ignore whole directories in .airflowignore
> --
>
> Key: AIRFLOW-1729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1729
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: Airflow 2.0
>Reporter: Cedric Hourcade
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 1.10.0
>
>
> The .airflowignore file allows to prevent scanning files for DAG. But even if
> we blacklist fulldirectory the {{os.walk}} will still go through them no
> matter how deep they are and skip files one by one, which can be an issue
> when you keep around big .git or virtualvenv directories.
> I suggest to add something like:
> {code}
> dirs[:] = [d for d in dirs if not any([re.findall(p, os.path.join(root, d))
> for p in patterns])]
> {code}
> to prune the directories here:
> https://github.com/apache/incubator-airflow/blob/cfc2f73c445074e1e09d6ef6a056cd2b33a945da/airflow/utils/dag_processing.py#L208-L209
> and in {{list_py_file_paths}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-1729) Ignore whole directories in .airflowignore
[
https://issues.apache.org/jira/browse/AIRFLOW-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin closed AIRFLOW-1729.
---
Resolution: Fixed
> Ignore whole directories in .airflowignore
> --
>
> Key: AIRFLOW-1729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1729
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: Airflow 2.0
>Reporter: Cedric Hourcade
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 1.10.0
>
>
> The .airflowignore file allows to prevent scanning files for DAG. But even if
> we blacklist fulldirectory the {{os.walk}} will still go through them no
> matter how deep they are and skip files one by one, which can be an issue
> when you keep around big .git or virtualvenv directories.
> I suggest to add something like:
> {code}
> dirs[:] = [d for d in dirs if not any([re.findall(p, os.path.join(root, d))
> for p in patterns])]
> {code}
> to prune the directories here:
> https://github.com/apache/incubator-airflow/blob/cfc2f73c445074e1e09d6ef6a056cd2b33a945da/airflow/utils/dag_processing.py#L208-L209
> and in {{list_py_file_paths}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1729) Ignore whole directories in .airflowignore
[
https://issues.apache.org/jira/browse/AIRFLOW-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542905#comment-16542905
]
ASF subversion and git services commented on AIRFLOW-1729:
--
Commit 23191605e463c85f0935cbf5c47f31c357d1596b in incubator-airflow's branch
refs/heads/v1-10-test from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=2319160 ]
[AIRFLOW-1729][AIRFLOW-2797][AIRFLOW-2729] Ignore whole directories in
.airflowignore
We can ignore whole directories by removing them
from the `dirs` array
that `os.walk()` returns. Doing this means that we
fewer disk ops if
someone has a set of modules in their dag folder
that they want to
ignore.
Also fixes [AIRFLOW-2797] - we weren't honoring
.airflowignore from a
parent dir as of #3717 -- that (expected)
behaviour is now back again.
De-duplicate the walking code as well - we had two
versions that had
gotten out of sync as of #3171. So that doesn't
happen again we now only
have one version.
Closes #3602 from ashb/ignore-whole-dirs-
airflowignore
(cherry picked from commit 6b2fdbef0ab4bd1ed91e6338bcf6440e782b7035)
Signed-off-by: Bolke de Bruin
> Ignore whole directories in .airflowignore
> --
>
> Key: AIRFLOW-1729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1729
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: Airflow 2.0
>Reporter: Cedric Hourcade
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 2.0.0
>
>
> The .airflowignore file allows to prevent scanning files for DAG. But even if
> we blacklist fulldirectory the {{os.walk}} will still go through them no
> matter how deep they are and skip files one by one, which can be an issue
> when you keep around big .git or virtualvenv directories.
> I suggest to add something like:
> {code}
> dirs[:] = [d for d in dirs if not any([re.findall(p, os.path.join(root, d))
> for p in patterns])]
> {code}
> to prune the directories here:
> https://github.com/apache/incubator-airflow/blob/cfc2f73c445074e1e09d6ef6a056cd2b33a945da/airflow/utils/dag_processing.py#L208-L209
> and in {{list_py_file_paths}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2729) .airflowignore is not being respected
[
https://issues.apache.org/jira/browse/AIRFLOW-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542906#comment-16542906
]
ASF subversion and git services commented on AIRFLOW-2729:
--
Commit 23191605e463c85f0935cbf5c47f31c357d1596b in incubator-airflow's branch
refs/heads/v1-10-test from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=2319160 ]
[AIRFLOW-1729][AIRFLOW-2797][AIRFLOW-2729] Ignore whole directories in
.airflowignore
We can ignore whole directories by removing them
from the `dirs` array
that `os.walk()` returns. Doing this means that we
fewer disk ops if
someone has a set of modules in their dag folder
that they want to
ignore.
Also fixes [AIRFLOW-2797] - we weren't honoring
.airflowignore from a
parent dir as of #3717 -- that (expected)
behaviour is now back again.
De-duplicate the walking code as well - we had two
versions that had
gotten out of sync as of #3171. So that doesn't
happen again we now only
have one version.
Closes #3602 from ashb/ignore-whole-dirs-
airflowignore
(cherry picked from commit 6b2fdbef0ab4bd1ed91e6338bcf6440e782b7035)
Signed-off-by: Bolke de Bruin
> .airflowignore is not being respected
> -
>
> Key: AIRFLOW-2729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2729
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10
>Reporter: James Meickle
>Assignee: Ash Berlin-Taylor
>Priority: Minor
>
> I have a repo that in 1.10 is giving airflowignore errors that did not exist
> in 1.9. I have a DAG repo with the following .airflowignore:
> {{airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/}}
> {{total 172}}
> {{drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .}}
> {{drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..}}
> {{-rw-r--r-- 1 airflow airflow 13 Jul 9 16:20 .airflowignore}}
> {{airflow@airflow-core-i-063df3268720e58fd:~$ cat
> /home/airflow/airflow/dags/airflow-tasks/.airflowignore}}
> {{submodules/*}}
> However, the submoduled repository is being scanned for DAGs anyways,
> including the test suite. Note the paths in the section below:
>
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,814]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/quantflow/operators/zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,817]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,818]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py",
> line 6, in }}
> {{Jul 09 18:52:01 airflow_web-stdout.log: from moto import mock_sqs}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named
> 'moto'}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,821]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,822]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 air
[jira] [Commented] (AIRFLOW-2729) .airflowignore is not being respected
[
https://issues.apache.org/jira/browse/AIRFLOW-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542904#comment-16542904
]
ASF subversion and git services commented on AIRFLOW-2729:
--
Commit 6b2fdbef0ab4bd1ed91e6338bcf6440e782b7035 in incubator-airflow's branch
refs/heads/master from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6b2fdbe ]
[AIRFLOW-1729][AIRFLOW-2797][AIRFLOW-2729] Ignore whole directories in
.airflowignore
We can ignore whole directories by removing them
from the `dirs` array
that `os.walk()` returns. Doing this means that we
fewer disk ops if
someone has a set of modules in their dag folder
that they want to
ignore.
Also fixes [AIRFLOW-2797] - we weren't honoring
.airflowignore from a
parent dir as of #3717 -- that (expected)
behaviour is now back again.
De-duplicate the walking code as well - we had two
versions that had
gotten out of sync as of #3171. So that doesn't
happen again we now only
have one version.
Closes #3602 from ashb/ignore-whole-dirs-
airflowignore
> .airflowignore is not being respected
> -
>
> Key: AIRFLOW-2729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2729
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10
>Reporter: James Meickle
>Assignee: Ash Berlin-Taylor
>Priority: Minor
>
> I have a repo that in 1.10 is giving airflowignore errors that did not exist
> in 1.9. I have a DAG repo with the following .airflowignore:
> {{airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/}}
> {{total 172}}
> {{drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .}}
> {{drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..}}
> {{-rw-r--r-- 1 airflow airflow 13 Jul 9 16:20 .airflowignore}}
> {{airflow@airflow-core-i-063df3268720e58fd:~$ cat
> /home/airflow/airflow/dags/airflow-tasks/.airflowignore}}
> {{submodules/*}}
> However, the submoduled repository is being scanned for DAGs anyways,
> including the test suite. Note the paths in the section below:
>
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,814]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/quantflow/operators/zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,817]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,818]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py",
> line 6, in }}
> {{Jul 09 18:52:01 airflow_web-stdout.log: from moto import mock_sqs}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named
> 'moto'}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,821]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,822]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.
[jira] [Commented] (AIRFLOW-1729) Ignore whole directories in .airflowignore
[
https://issues.apache.org/jira/browse/AIRFLOW-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542902#comment-16542902
]
ASF subversion and git services commented on AIRFLOW-1729:
--
Commit 6b2fdbef0ab4bd1ed91e6338bcf6440e782b7035 in incubator-airflow's branch
refs/heads/master from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6b2fdbe ]
[AIRFLOW-1729][AIRFLOW-2797][AIRFLOW-2729] Ignore whole directories in
.airflowignore
We can ignore whole directories by removing them
from the `dirs` array
that `os.walk()` returns. Doing this means that we
fewer disk ops if
someone has a set of modules in their dag folder
that they want to
ignore.
Also fixes [AIRFLOW-2797] - we weren't honoring
.airflowignore from a
parent dir as of #3717 -- that (expected)
behaviour is now back again.
De-duplicate the walking code as well - we had two
versions that had
gotten out of sync as of #3171. So that doesn't
happen again we now only
have one version.
Closes #3602 from ashb/ignore-whole-dirs-
airflowignore
> Ignore whole directories in .airflowignore
> --
>
> Key: AIRFLOW-1729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1729
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: Airflow 2.0
>Reporter: Cedric Hourcade
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 2.0.0
>
>
> The .airflowignore file allows to prevent scanning files for DAG. But even if
> we blacklist fulldirectory the {{os.walk}} will still go through them no
> matter how deep they are and skip files one by one, which can be an issue
> when you keep around big .git or virtualvenv directories.
> I suggest to add something like:
> {code}
> dirs[:] = [d for d in dirs if not any([re.findall(p, os.path.join(root, d))
> for p in patterns])]
> {code}
> to prune the directories here:
> https://github.com/apache/incubator-airflow/blob/cfc2f73c445074e1e09d6ef6a056cd2b33a945da/airflow/utils/dag_processing.py#L208-L209
> and in {{list_py_file_paths}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-1729][AIRFLOW-2797][AIRFLOW-2729] Ignore whole directories in .airflowignore
Repository: incubator-airflow
Updated Branches:
refs/heads/v1-10-test f4bcc3356 -> 23191605e
[AIRFLOW-1729][AIRFLOW-2797][AIRFLOW-2729] Ignore whole directories in
.airflowignore
We can ignore whole directories by removing them
from the `dirs` array
that `os.walk()` returns. Doing this means that we
fewer disk ops if
someone has a set of modules in their dag folder
that they want to
ignore.
Also fixes [AIRFLOW-2797] - we weren't honoring
.airflowignore from a
parent dir as of #3717 -- that (expected)
behaviour is now back again.
De-duplicate the walking code as well - we had two
versions that had
gotten out of sync as of #3171. So that doesn't
happen again we now only
have one version.
Closes #3602 from ashb/ignore-whole-dirs-
airflowignore
(cherry picked from commit 6b2fdbef0ab4bd1ed91e6338bcf6440e782b7035)
Signed-off-by: Bolke de Bruin
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/23191605
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/23191605
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/23191605
Branch: refs/heads/v1-10-test
Commit: 23191605e463c85f0935cbf5c47f31c357d1596b
Parents: f4bcc33
Author: Ash Berlin-Taylor
Authored: Fri Jul 13 13:18:47 2018 +0200
Committer: Bolke de Bruin
Committed: Fri Jul 13 13:19:08 2018 +0200
--
airflow/models.py | 58 +---
airflow/utils/dag_processing.py | 29 ++
2 files changed, 44 insertions(+), 43 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/23191605/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index bde6137..5f0713f 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -510,7 +510,8 @@ class DagBag(BaseDagBag, LoggingMixin):
Note that if a .airflowignore file is found while processing,
the directory, it will behaves much like a .gitignore does,
ignoring files that match any of the regex patterns specified
-in the file.
+in the file. **Note**: The patterns in .airflowignore are treated as
+un-anchored regexes, not shell-like glob patterns.
"""
start_dttm = timezone.utcnow()
dag_folder = dag_folder or self.dag_folder
@@ -519,42 +520,25 @@ class DagBag(BaseDagBag, LoggingMixin):
stats = []
FileLoadStat = namedtuple(
'FileLoadStat', "file duration dag_num task_num dags")
-if os.path.isfile(dag_folder):
-self.process_file(dag_folder, only_if_updated=only_if_updated)
-elif os.path.isdir(dag_folder):
-for root, dirs, files in os.walk(dag_folder, followlinks=True):
-patterns = []
-ignore_file = os.path.join(root, '.airflowignore')
-if os.path.isfile(ignore_file):
-with open(ignore_file, 'r') as f:
-patterns += [p for p in f.read().split('\n') if p]
-for f in files:
-try:
-filepath = os.path.join(root, f)
-if not os.path.isfile(filepath):
-continue
-mod_name, file_ext = os.path.splitext(
-os.path.split(filepath)[-1])
-if file_ext != '.py' and not
zipfile.is_zipfile(filepath):
-continue
-if not any(
-[re.findall(p, filepath) for p in patterns]):
-ts = timezone.utcnow()
-found_dags = self.process_file(
-filepath, only_if_updated=only_if_updated)
-
-td = timezone.utcnow() - ts
-td = td.total_seconds() + (
-float(td.microseconds) / 100)
-stats.append(FileLoadStat(
-filepath.replace(dag_folder, ''),
-td,
-len(found_dags),
-sum([len(dag.tasks) for dag in found_dags]),
-str([dag.dag_id for dag in found_dags]),
-))
-except Exception as e:
-self.log.exception(e)
+for filepath in utils.dag_processing.list_py_file_paths(dag_folder):
+self.log.info(filepath)
+try:
+ts = timezone.utcnow()
+found_dags = self.process_file(
+filepath, only_if_updated=only_if_updated)
+
+td = ti
incubator-airflow git commit: [AIRFLOW-1729][AIRFLOW-2797][AIRFLOW-2729] Ignore whole directories in .airflowignore
Repository: incubator-airflow
Updated Branches:
refs/heads/master 5290688ee -> 6b2fdbef0
[AIRFLOW-1729][AIRFLOW-2797][AIRFLOW-2729] Ignore whole directories in
.airflowignore
We can ignore whole directories by removing them
from the `dirs` array
that `os.walk()` returns. Doing this means that we
fewer disk ops if
someone has a set of modules in their dag folder
that they want to
ignore.
Also fixes [AIRFLOW-2797] - we weren't honoring
.airflowignore from a
parent dir as of #3717 -- that (expected)
behaviour is now back again.
De-duplicate the walking code as well - we had two
versions that had
gotten out of sync as of #3171. So that doesn't
happen again we now only
have one version.
Closes #3602 from ashb/ignore-whole-dirs-
airflowignore
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6b2fdbef
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6b2fdbef
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6b2fdbef
Branch: refs/heads/master
Commit: 6b2fdbef0ab4bd1ed91e6338bcf6440e782b7035
Parents: 5290688
Author: Ash Berlin-Taylor
Authored: Fri Jul 13 13:18:47 2018 +0200
Committer: Bolke de Bruin
Committed: Fri Jul 13 13:18:47 2018 +0200
--
airflow/models.py | 58 +---
airflow/utils/dag_processing.py | 29 ++
2 files changed, 44 insertions(+), 43 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6b2fdbef/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index 089befe..21df933 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -510,7 +510,8 @@ class DagBag(BaseDagBag, LoggingMixin):
Note that if a .airflowignore file is found while processing,
the directory, it will behaves much like a .gitignore does,
ignoring files that match any of the regex patterns specified
-in the file.
+in the file. **Note**: The patterns in .airflowignore are treated as
+un-anchored regexes, not shell-like glob patterns.
"""
start_dttm = timezone.utcnow()
dag_folder = dag_folder or self.dag_folder
@@ -519,42 +520,25 @@ class DagBag(BaseDagBag, LoggingMixin):
stats = []
FileLoadStat = namedtuple(
'FileLoadStat', "file duration dag_num task_num dags")
-if os.path.isfile(dag_folder):
-self.process_file(dag_folder, only_if_updated=only_if_updated)
-elif os.path.isdir(dag_folder):
-for root, dirs, files in os.walk(dag_folder, followlinks=True):
-patterns = []
-ignore_file = os.path.join(root, '.airflowignore')
-if os.path.isfile(ignore_file):
-with open(ignore_file, 'r') as f:
-patterns += [p for p in f.read().split('\n') if p]
-for f in files:
-try:
-filepath = os.path.join(root, f)
-if not os.path.isfile(filepath):
-continue
-mod_name, file_ext = os.path.splitext(
-os.path.split(filepath)[-1])
-if file_ext != '.py' and not
zipfile.is_zipfile(filepath):
-continue
-if not any(
-[re.findall(p, filepath) for p in patterns]):
-ts = timezone.utcnow()
-found_dags = self.process_file(
-filepath, only_if_updated=only_if_updated)
-
-td = timezone.utcnow() - ts
-td = td.total_seconds() + (
-float(td.microseconds) / 100)
-stats.append(FileLoadStat(
-filepath.replace(dag_folder, ''),
-td,
-len(found_dags),
-sum([len(dag.tasks) for dag in found_dags]),
-str([dag.dag_id for dag in found_dags]),
-))
-except Exception as e:
-self.log.exception(e)
+for filepath in utils.dag_processing.list_py_file_paths(dag_folder):
+self.log.info(filepath)
+try:
+ts = timezone.utcnow()
+found_dags = self.process_file(
+filepath, only_if_updated=only_if_updated)
+
+td = timezone.utcnow() - ts
+td = td.total_seconds() + (
+float(td.microseconds)
[jira] [Resolved] (AIRFLOW-2723) Update lxml package for python 3 compatibility
[ https://issues.apache.org/jira/browse/AIRFLOW-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-2723. --- Resolution: Fixed Fix Version/s: (was: Airflow 1.9.0) (was: Airflow 2.0) 2.0.0 Issue resolved by pull request #3583 [https://github.com/apache/incubator-airflow/pull/3583] > Update lxml package for python 3 compatibility > -- > > Key: AIRFLOW-2723 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2723 > Project: Apache Airflow > Issue Type: Bug > Components: dependencies, docker >Affects Versions: Airflow 2.0, Airflow 1.9.0 >Reporter: neil90 >Assignee: neil90 >Priority: Minor > Fix For: 2.0.0 > > > Following the > [CONTRIBUTING.md|https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#set-up-a-development-env-using-docker] > for testing env for python 3 in docker I reviewed a gcc when pip install > position failed on the package gcc. Link to error - > [https://pastebin.com/30AJz5tC] . > > I propose changing the setup.py lxml install to version 4.0.0 or greater. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2723) Update lxml package for python 3 compatibility
[ https://issues.apache.org/jira/browse/AIRFLOW-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542833#comment-16542833 ] ASF subversion and git services commented on AIRFLOW-2723: -- Commit 5290688ee0576ad167d9622c96cdeb08e9965a20 in incubator-airflow's branch refs/heads/master from [~neilp90] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=5290688 ] [AIRFLOW-2723] Update lxml dependancy to >= 4.0.0 Closes #3583 from neil90/master > Update lxml package for python 3 compatibility > -- > > Key: AIRFLOW-2723 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2723 > Project: Apache Airflow > Issue Type: Bug > Components: dependencies, docker >Affects Versions: Airflow 2.0, Airflow 1.9.0 >Reporter: neil90 >Assignee: neil90 >Priority: Minor > Fix For: Airflow 2.0, Airflow 1.9.0 > > > Following the > [CONTRIBUTING.md|https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#set-up-a-development-env-using-docker] > for testing env for python 3 in docker I reviewed a gcc when pip install > position failed on the package gcc. Link to error - > [https://pastebin.com/30AJz5tC] . > > I propose changing the setup.py lxml install to version 4.0.0 or greater. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2723] Update lxml dependancy to >= 4.0.0
Repository: incubator-airflow Updated Branches: refs/heads/master 78da52fee -> 5290688ee [AIRFLOW-2723] Update lxml dependancy to >= 4.0.0 Closes #3583 from neil90/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/5290688e Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/5290688e Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/5290688e Branch: refs/heads/master Commit: 5290688ee0576ad167d9622c96cdeb08e9965a20 Parents: 78da52f Author: Neil Patel Authored: Fri Jul 13 12:12:39 2018 +0200 Committer: Fokko Driesprong Committed: Fri Jul 13 12:12:39 2018 +0200 -- setup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/5290688e/setup.py -- diff --git a/setup.py b/setup.py index 99a31ad..630c793 100644 --- a/setup.py +++ b/setup.py @@ -204,7 +204,7 @@ devel = [ 'click', 'freezegun', 'jira', -'lxml>=3.3.4', +'lxml>=4.0.0', 'mock', 'mongomock', 'moto==1.1.19', @@ -270,7 +270,7 @@ def do_setup(): 'gunicorn>=19.4.0, <20.0', 'iso8601>=0.1.12', 'jinja2>=2.7.3, <2.9.0', -'lxml>=3.6.0, <4.0', +'lxml>=4.0.0', 'markdown>=2.5.2, <3.0', 'pandas>=0.17.1, <1.0.0', 'pendulum==1.4.4',
[jira] [Resolved] (AIRFLOW-2739) Airflow crashes on startup if LC_ALL env isnt set to utf-8
[
https://issues.apache.org/jira/browse/AIRFLOW-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin resolved AIRFLOW-2739.
-
Resolution: Fixed
Fix Version/s: 1.10.0
Issue resolved by pull request #3593
[https://github.com/apache/incubator-airflow/pull/3593]
> Airflow crashes on startup if LC_ALL env isnt set to utf-8
> --
>
> Key: AIRFLOW-2739
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2739
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.10.0
> Environment: Python 3.6.0, Ubuntu 14.04.5 LTS
>Reporter: Carl Johan Gustavsson
>Assignee: Carl Johan Gustavsson
>Priority: Major
> Fix For: 1.10.0
>
>
> When running Airflow 1.10.0 RC1 without LC_ALL environment variable set
> Airflow crashes on start with the following trace
>
> {code:java}
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback (most
> recent call last):
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
> line 35, in
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration as conf
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
> line 106, in
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 DEFAULT_CONFIG
> = f.read()
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line 26,
> in decode
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
> codecs.ascii_decode(input, self.errors)[0]
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
> ordinal not in range(128)
> {code}
> This is because the `config_templates/default_airflow.cfg` contains a
> non-ascii character and if LC_ALL isnt set to
> `{color:#00}en_US.UTF-8{color}` or similar Python will assume the file is
> in ascii.
>
> Solution would be to always open the config files as utf-8 regardless of the
> LC_ALL environment variable.
>
> This worked up until
> [https://github.com/apache/incubator-airflow/commit/16bae5634df24132b37eb752fe816f51bf7e83ca]
> it seems.
>
> Python versions affected, 3.4.0, 3.5.5, 3.6.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2739) Airflow crashes on startup if LC_ALL env isnt set to utf-8
[
https://issues.apache.org/jira/browse/AIRFLOW-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542811#comment-16542811
]
ASF subversion and git services commented on AIRFLOW-2739:
--
Commit f4bcc33564d68b2146fb4fbf6a08f20326069306 in incubator-airflow's branch
refs/heads/v1-10-test from [~cjg]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f4bcc33 ]
[AIRFLOW-2739] Always read default configuration files as utf-8
Closes #3593 from cjgu/airflow-2739
(cherry picked from commit 78da52fee477b0f7b2ee3481d32a06fca7a9841c)
Signed-off-by: Bolke de Bruin
> Airflow crashes on startup if LC_ALL env isnt set to utf-8
> --
>
> Key: AIRFLOW-2739
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2739
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.10.0
> Environment: Python 3.6.0, Ubuntu 14.04.5 LTS
>Reporter: Carl Johan Gustavsson
>Assignee: Carl Johan Gustavsson
>Priority: Major
> Fix For: 1.10.0
>
>
> When running Airflow 1.10.0 RC1 without LC_ALL environment variable set
> Airflow crashes on start with the following trace
>
> {code:java}
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback (most
> recent call last):
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
> line 35, in
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration as conf
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
> line 106, in
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 DEFAULT_CONFIG
> = f.read()
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line 26,
> in decode
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
> codecs.ascii_decode(input, self.errors)[0]
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
> ordinal not in range(128)
> {code}
> This is because the `config_templates/default_airflow.cfg` contains a
> non-ascii character and if LC_ALL isnt set to
> `{color:#00}en_US.UTF-8{color}` or similar Python will assume the file is
> in ascii.
>
> Solution would be to always open the config files as utf-8 regardless of the
> LC_ALL environment variable.
>
> This worked up until
> [https://github.com/apache/incubator-airflow/commit/16bae5634df24132b37eb752fe816f51bf7e83ca]
> it seems.
>
> Python versions affected, 3.4.0, 3.5.5, 3.6.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2739) Airflow crashes on startup if LC_ALL env isnt set to utf-8
[
https://issues.apache.org/jira/browse/AIRFLOW-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542810#comment-16542810
]
ASF subversion and git services commented on AIRFLOW-2739:
--
Commit 78da52fee477b0f7b2ee3481d32a06fca7a9841c in incubator-airflow's branch
refs/heads/master from [~cjg]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=78da52f ]
[AIRFLOW-2739] Always read default configuration files as utf-8
Closes #3593 from cjgu/airflow-2739
> Airflow crashes on startup if LC_ALL env isnt set to utf-8
> --
>
> Key: AIRFLOW-2739
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2739
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.10.0
> Environment: Python 3.6.0, Ubuntu 14.04.5 LTS
>Reporter: Carl Johan Gustavsson
>Assignee: Carl Johan Gustavsson
>Priority: Major
>
> When running Airflow 1.10.0 RC1 without LC_ALL environment variable set
> Airflow crashes on start with the following trace
>
> {code:java}
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback (most
> recent call last):
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
> line 35, in
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration as conf
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
> line 106, in
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 DEFAULT_CONFIG
> = f.read()
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line 26,
> in decode
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
> codecs.ascii_decode(input, self.errors)[0]
> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
> ordinal not in range(128)
> {code}
> This is because the `config_templates/default_airflow.cfg` contains a
> non-ascii character and if LC_ALL isnt set to
> `{color:#00}en_US.UTF-8{color}` or similar Python will assume the file is
> in ascii.
>
> Solution would be to always open the config files as utf-8 regardless of the
> LC_ALL environment variable.
>
> This worked up until
> [https://github.com/apache/incubator-airflow/commit/16bae5634df24132b37eb752fe816f51bf7e83ca]
> it seems.
>
> Python versions affected, 3.4.0, 3.5.5, 3.6.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2739] Always read default configuration files as utf-8
Repository: incubator-airflow
Updated Branches:
refs/heads/v1-10-test f123cf5f8 -> f4bcc3356
[AIRFLOW-2739] Always read default configuration files as utf-8
Closes #3593 from cjgu/airflow-2739
(cherry picked from commit 78da52fee477b0f7b2ee3481d32a06fca7a9841c)
Signed-off-by: Bolke de Bruin
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f4bcc335
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f4bcc335
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f4bcc335
Branch: refs/heads/v1-10-test
Commit: f4bcc33564d68b2146fb4fbf6a08f20326069306
Parents: f123cf5
Author: Carl Johan Gustavsson
Authored: Fri Jul 13 11:58:39 2018 +0200
Committer: Bolke de Bruin
Committed: Fri Jul 13 11:58:53 2018 +0200
--
airflow/configuration.py | 24 ++--
1 file changed, 14 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f4bcc335/airflow/configuration.py
--
diff --git a/airflow/configuration.py b/airflow/configuration.py
index e2089e5..2e05fde 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -101,15 +101,20 @@ def run_command(command):
return output
-_templates_dir = os.path.join(os.path.dirname(__file__), 'config_templates')
-with open(os.path.join(_templates_dir, 'default_airflow.cfg')) as f:
-DEFAULT_CONFIG = f.read()
+def _read_default_config_file(file_name):
+templates_dir = os.path.join(os.path.dirname(__file__), 'config_templates')
+file_path = os.path.join(templates_dir, file_name)
if six.PY2:
-DEFAULT_CONFIG = DEFAULT_CONFIG.decode('utf-8')
-with open(os.path.join(_templates_dir, 'default_test.cfg')) as f:
-TEST_CONFIG = f.read()
-if six.PY2:
-TEST_CONFIG = TEST_CONFIG.decode('utf-8')
+with open(file_path) as f:
+config = f.read()
+return config.decode('utf-8')
+else:
+with open(file_path, encoding='utf-8') as f:
+return f.read()
+
+
+DEFAULT_CONFIG = _read_default_config_file('default_airflow.cfg')
+TEST_CONFIG = _read_default_config_file('default_test.cfg')
class AirflowConfigParser(ConfigParser):
@@ -502,8 +507,7 @@ conf.read(AIRFLOW_CONFIG)
if conf.getboolean('webserver', 'rbac'):
-with open(os.path.join(_templates_dir, 'default_webserver_config.py')) as
f:
-DEFAULT_WEBSERVER_CONFIG = f.read()
+DEFAULT_WEBSERVER_CONFIG =
_read_default_config_file('default_webserver_config.py')
WEBSERVER_CONFIG = AIRFLOW_HOME + '/webserver_config.py'
incubator-airflow git commit: [AIRFLOW-2739] Always read default configuration files as utf-8
Repository: incubator-airflow
Updated Branches:
refs/heads/master 39adc77f6 -> 78da52fee
[AIRFLOW-2739] Always read default configuration files as utf-8
Closes #3593 from cjgu/airflow-2739
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/78da52fe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/78da52fe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/78da52fe
Branch: refs/heads/master
Commit: 78da52fee477b0f7b2ee3481d32a06fca7a9841c
Parents: 39adc77
Author: Carl Johan Gustavsson
Authored: Fri Jul 13 11:58:39 2018 +0200
Committer: Bolke de Bruin
Committed: Fri Jul 13 11:58:39 2018 +0200
--
airflow/configuration.py | 24 ++--
1 file changed, 14 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/78da52fe/airflow/configuration.py
--
diff --git a/airflow/configuration.py b/airflow/configuration.py
index e2089e5..2e05fde 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -101,15 +101,20 @@ def run_command(command):
return output
-_templates_dir = os.path.join(os.path.dirname(__file__), 'config_templates')
-with open(os.path.join(_templates_dir, 'default_airflow.cfg')) as f:
-DEFAULT_CONFIG = f.read()
+def _read_default_config_file(file_name):
+templates_dir = os.path.join(os.path.dirname(__file__), 'config_templates')
+file_path = os.path.join(templates_dir, file_name)
if six.PY2:
-DEFAULT_CONFIG = DEFAULT_CONFIG.decode('utf-8')
-with open(os.path.join(_templates_dir, 'default_test.cfg')) as f:
-TEST_CONFIG = f.read()
-if six.PY2:
-TEST_CONFIG = TEST_CONFIG.decode('utf-8')
+with open(file_path) as f:
+config = f.read()
+return config.decode('utf-8')
+else:
+with open(file_path, encoding='utf-8') as f:
+return f.read()
+
+
+DEFAULT_CONFIG = _read_default_config_file('default_airflow.cfg')
+TEST_CONFIG = _read_default_config_file('default_test.cfg')
class AirflowConfigParser(ConfigParser):
@@ -502,8 +507,7 @@ conf.read(AIRFLOW_CONFIG)
if conf.getboolean('webserver', 'rbac'):
-with open(os.path.join(_templates_dir, 'default_webserver_config.py')) as
f:
-DEFAULT_WEBSERVER_CONFIG = f.read()
+DEFAULT_WEBSERVER_CONFIG =
_read_default_config_file('default_webserver_config.py')
WEBSERVER_CONFIG = AIRFLOW_HOME + '/webserver_config.py'
[2/2] incubator-airflow git commit: [AIRFLOW-XXX] Use .sha512 for signature
[AIRFLOW-XXX] Use .sha512 for signature
As requested by the IPMC
(cherry picked from commit a54f74821e161fd9e5423f5cfebaeab43a021422)
Signed-off-by: Bolke de Bruin
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f123cf5f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f123cf5f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f123cf5f
Branch: refs/heads/v1-10-test
Commit: f123cf5f8de71152d1b2b39c5c994bfc00a67d72
Parents: 8e54a29
Author: Bolke de Bruin
Authored: Fri Jul 13 11:42:41 2018 +0200
Committer: Bolke de Bruin
Committed: Fri Jul 13 11:55:28 2018 +0200
--
dev/sign.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f123cf5f/dev/sign.sh
--
diff --git a/dev/sign.sh b/dev/sign.sh
index da0f7a9..66427a7 100755
--- a/dev/sign.sh
+++ b/dev/sign.sh
@@ -25,5 +25,5 @@
NAME=${1}
gpg --armor --output ${NAME}.asc --detach-sig ${NAME}
-gpg --print-md SHA512 ${NAME} > ${NAME}.sha
+gpg --print-md SHA512 ${NAME} > ${NAME}.sha512
gpg --print-md MD5 ${NAME} > ${NAME}.md5
[1/2] incubator-airflow git commit: [AIRFLOW-XXX] Wrap DISCLAIMER text
Repository: incubator-airflow Updated Branches: refs/heads/v1-10-test f1083cbad -> f123cf5f8 [AIRFLOW-XXX] Wrap DISCLAIMER text (cherry picked from commit 39adc77f6fefc54fcfa893362b576070a4c3e660) Signed-off-by: Bolke de Bruin Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8e54a291 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8e54a291 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8e54a291 Branch: refs/heads/v1-10-test Commit: 8e54a2917ef86d5cbdd8533a4fd4f38eaf6cc144 Parents: f1083cb Author: Bolke de Bruin Authored: Fri Jul 13 11:46:03 2018 +0200 Committer: Bolke de Bruin Committed: Fri Jul 13 11:54:56 2018 +0200 -- DISCLAIMER | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8e54a291/DISCLAIMER -- diff --git a/DISCLAIMER b/DISCLAIMER index 8fe6988..2758508 100644 --- a/DISCLAIMER +++ b/DISCLAIMER @@ -1 +1,6 @@ -Apache Airflow is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. +Apache Airflow is an effort undergoing incubation at The Apache Software Foundation (ASF), +sponsored by the Apache Incubator. Incubation is required of all newly accepted projects +until a further review indicates that the infrastructure, communications, and decision +making process have stabilized in a manner consistent with other successful ASF projects. +While incubation status is not necessarily a reflection of the completeness or stability +of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
[1/2] incubator-airflow git commit: [AIRFLOW-XXX] Use .sha512 for signature
Repository: incubator-airflow
Updated Branches:
refs/heads/master aa17b5baf -> 39adc77f6
[AIRFLOW-XXX] Use .sha512 for signature
As requested by the IPMC
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a54f7482
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a54f7482
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a54f7482
Branch: refs/heads/master
Commit: a54f74821e161fd9e5423f5cfebaeab43a021422
Parents: aa17b5b
Author: Bolke de Bruin
Authored: Fri Jul 13 11:42:41 2018 +0200
Committer: Bolke de Bruin
Committed: Fri Jul 13 11:42:41 2018 +0200
--
dev/sign.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a54f7482/dev/sign.sh
--
diff --git a/dev/sign.sh b/dev/sign.sh
index da0f7a9..66427a7 100755
--- a/dev/sign.sh
+++ b/dev/sign.sh
@@ -25,5 +25,5 @@
NAME=${1}
gpg --armor --output ${NAME}.asc --detach-sig ${NAME}
-gpg --print-md SHA512 ${NAME} > ${NAME}.sha
+gpg --print-md SHA512 ${NAME} > ${NAME}.sha512
gpg --print-md MD5 ${NAME} > ${NAME}.md5
[2/2] incubator-airflow git commit: [AIRFLOW-XXX] Wrap DISCLAIMER text
[AIRFLOW-XXX] Wrap DISCLAIMER text Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/39adc77f Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/39adc77f Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/39adc77f Branch: refs/heads/master Commit: 39adc77f6fefc54fcfa893362b576070a4c3e660 Parents: a54f748 Author: Bolke de Bruin Authored: Fri Jul 13 11:46:03 2018 +0200 Committer: Bolke de Bruin Committed: Fri Jul 13 11:46:03 2018 +0200 -- DISCLAIMER | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/39adc77f/DISCLAIMER -- diff --git a/DISCLAIMER b/DISCLAIMER index 8fe6988..2758508 100644 --- a/DISCLAIMER +++ b/DISCLAIMER @@ -1 +1,6 @@ -Apache Airflow is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. +Apache Airflow is an effort undergoing incubation at The Apache Software Foundation (ASF), +sponsored by the Apache Incubator. Incubation is required of all newly accepted projects +until a further review indicates that the infrastructure, communications, and decision +making process have stabilized in a manner consistent with other successful ASF projects. +While incubation status is not necessarily a reflection of the completeness or stability +of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
[jira] [Assigned] (AIRFLOW-2729) .airflowignore is not being respected
[
https://issues.apache.org/jira/browse/AIRFLOW-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor reassigned AIRFLOW-2729:
--
Assignee: Ash Berlin-Taylor
External issue URL: https://github.com/apache/incubator-airflow/pull/3602
> .airflowignore is not being respected
> -
>
> Key: AIRFLOW-2729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2729
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10
>Reporter: James Meickle
>Assignee: Ash Berlin-Taylor
>Priority: Minor
>
> I have a repo that in 1.10 is giving airflowignore errors that did not exist
> in 1.9. I have a DAG repo with the following .airflowignore:
> {{airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/}}
> {{total 172}}
> {{drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .}}
> {{drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..}}
> {{-rw-r--r-- 1 airflow airflow 13 Jul 9 16:20 .airflowignore}}
> {{airflow@airflow-core-i-063df3268720e58fd:~$ cat
> /home/airflow/airflow/dags/airflow-tasks/.airflowignore}}
> {{submodules/*}}
> However, the submoduled repository is being scanned for DAGs anyways,
> including the test suite. Note the paths in the section below:
>
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,814]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/quantflow/operators/zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,817]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,818]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py",
> line 6, in }}
> {{Jul 09 18:52:01 airflow_web-stdout.log: from moto import mock_sqs}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named
> 'moto'}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,821]
> \{{models.py:351}} DEBUG - Importing
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,822]
> \{{models.py:365}} ERROR - Failed to import:
> /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py",
> line 362, in process_file}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name,
> filepath)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in
> load_source}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 693, in _load}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 673, in _load_unlocked}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap_external>", line 665, in exec_module}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File " importlib._bootstrap>", line 222, in _call_with_frames_removed}}
> {{Jul 09 18:52:01 airflow_web-stdout.log: File
> "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py",
> line 6, in }}
> {{Jul 09 18:52:01 airflow_web-stdout.l
[jira] [Assigned] (AIRFLOW-1729) Ignore whole directories in .airflowignore
[
https://issues.apache.org/jira/browse/AIRFLOW-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor reassigned AIRFLOW-1729:
--
Assignee: Ash Berlin-Taylor (was: Kamil Sambor)
External issue URL: https://github.com/apache/incubator-airflow/pull/3602
> Ignore whole directories in .airflowignore
> --
>
> Key: AIRFLOW-1729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1729
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: Airflow 2.0
>Reporter: Cedric Hourcade
>Assignee: Ash Berlin-Taylor
>Priority: Minor
> Fix For: 2.0.0
>
>
> The .airflowignore file allows to prevent scanning files for DAG. But even if
> we blacklist fulldirectory the {{os.walk}} will still go through them no
> matter how deep they are and skip files one by one, which can be an issue
> when you keep around big .git or virtualvenv directories.
> I suggest to add something like:
> {code}
> dirs[:] = [d for d in dirs if not any([re.findall(p, os.path.join(root, d))
> for p in patterns])]
> {code}
> to prune the directories here:
> https://github.com/apache/incubator-airflow/blob/cfc2f73c445074e1e09d6ef6a056cd2b33a945da/airflow/utils/dag_processing.py#L208-L209
> and in {{list_py_file_paths}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
