ashb commented on a change in pull request #7674: [AIRFLOW-7022] Simplify
DagFileProcessor.process_file method
URL: https://github.com/apache/airflow/pull/7674#discussion_r390259132
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -905,6 +898,28 @@ def process_file(
return simple_dags, len(dagbag.import_errors)
+ @provide_session
+ def _prepare_simple_dags(self, dags: List[DAG], pickle_dags: bool,
session) -> List[SimpleDag]:
+ """
+ Convert DAGS to SimpleDags. If necessary, it also Pickle the DAGs
+
+ :param dags: List of DAGs
+ :param pickle_dags: whether serialize the DAGs found in the file and
+ save them to the db
+ :type pickle_dags: bool
+ :return: List of SimpleDag
+ :rtype: List[airflow.utils.dag_processing.SimpleDag]
+ """
+
+ simple_dags = []
+ # Pickle the DAGs (if necessary) and put them into a SimpleDag
+ for dag in dags:
+ pickle_id = None
+ if pickle_dags:
+ pickle_id = dag.pickle(session).id
Review comment:
I'd say no to list comprehension here as it would be quite large because of
the `if pickle` condition/block. (And large list comprehensions are generally
more complex than it's worth for readability)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
