xinbinhuang commented on issue #7157: [AIRFLOW-6251] add config for max tasks per dag URL: https://github.com/apache/airflow/pull/7157#issuecomment-577052428 You can check the task count in the CI/CD process after deploying to source control similar to other DAG validation tests that you may have ([relevant blog post](https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c) & [gist](https://gist.github.com/criccomini/2862667822af7fae8b55682faef029a7). A simple example: ```python import unittest from airflow.models import DagBag class TestTaskCountPerDag(unittest.TestCase): """ Test the if the number of tasks per dag exceed the threshold """ MAX_TASKS_PER_DAG = 3 def _get_dagbag(self): dag_folder = os.getenv('AIRFLOW_DAGS', False) self.assertTrue( dag_folder, 'AIRFLOW_DAGS must be set to a folder that has DAGs in it.') return DagBag(dag_folder=dag_folder, include_examples=False) def test_max_tasks_per_dag(self): """ Verify that tasks per dag do not exceed the threshold """ dagbag = self._get_dagbag() dags = dagbag.dags for dag_name, dag in dags.items(): self.assertLessEqual(dag.task_count, MAX_TASKS_PER_DAG, msg=('Numer of tasks for DAG {} is greater than ' + 'the threshold {}').format(dag_name, MAX_TASKS_PER_DAG) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
