[
https://issues.apache.org/jira/browse/AIRFLOW-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544581#comment-16544581
]
ASF subversion and git services commented on AIRFLOW-1729:
----------------------------------------------------------
Commit fc00875ed00555e5da5661324479e8703787bbd5 in incubator-airflow's branch
refs/heads/v1-10-test from [~ashb]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fc00875 ]
[AIRFLOW-1729][AIRFLOW-XXX] Remove extra debug log at info level
I left an extra log call, at info level in #3602
that was being used for
debugging.
Closes #3603 from ashb/remove-extra-log
(cherry picked from commit 7c19ed02bc30f71fd73e38234b3b210072274f3b)
Signed-off-by: Bolke de Bruin <[email protected]>
> Ignore whole directories in .airflowignore
> ------------------------------------------
>
> Key: AIRFLOW-1729
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1729
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
> Affects Versions: Airflow 2.0
> Reporter: Cedric Hourcade
> Assignee: Ash Berlin-Taylor
> Priority: Minor
> Fix For: 1.10.0
>
>
> The .airflowignore file allows to prevent scanning files for DAG. But even if
> we blacklist fulldirectory the {{os.walk}} will still go through them no
> matter how deep they are and skip files one by one, which can be an issue
> when you keep around big .git or virtualvenv directories.
> I suggest to add something like:
> {code}
> dirs[:] = [d for d in dirs if not any([re.findall(p, os.path.join(root, d))
> for p in patterns])]
> {code}
> to prune the directories here:
> https://github.com/apache/incubator-airflow/blob/cfc2f73c445074e1e09d6ef6a056cd2b33a945da/airflow/utils/dag_processing.py#L208-L209
> and in {{list_py_file_paths}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
