ashb commented on a change in pull request #4746: [AIRFLOW-3932] Optionally
skip dag discovery heuristic.
URL: https://github.com/apache/airflow/pull/4746#discussion_r259269141
##########
File path: airflow/utils/dag_processing.py
##########
@@ -332,17 +332,18 @@ def list_py_file_paths(directory, safe_mode=True,
if any([re.findall(p, file_path) for p in patterns]):
continue
- # Heuristic that guesses whether a Python file contains an
- # Airflow DAG definition.
- might_contain_dag = True
- if safe_mode and not zipfile.is_zipfile(file_path):
- with open(file_path, 'rb') as f:
- content = f.read()
- might_contain_dag = all(
- [s in content for s in (b'DAG', b'airflow')])
-
- if not might_contain_dag:
- continue
+ if conf.getboolean('core', 'USE_DAG_DISCOVERY_HEURISTIC'):
Review comment:
Yes, except not like this. Pass the value of this config option in to the
pre-existing `safe_mode` parameter:
```
:param safe_mode: whether to use a heuristic to determine whether a file
contains Airflow DAG definitions
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
