potiuk commented on a change in pull request #5128: [AIRFLOW-4307] Backfill
respects concurrency limit
URL: https://github.com/apache/airflow/pull/5128#discussion_r277052157
##########
File path: tests/test_jobs.py
##########
@@ -308,6 +308,37 @@ def test_backfill_conf(self):
self.assertEqual(conf, dr[0].conf)
+ def test_backfill_respect_concurrency_limit(self):
+
+ dag = self._get_dummy_dag('test_backfill_respect_concurrency_limit')
+ dag.concurrency = 2
+
+ executor = TestExecutor(do_update=True)
+
+ job = BackfillJob(
+ dag=dag,
+ executor=executor,
+ start_date=DEFAULT_DATE,
+ end_date=DEFAULT_DATE + datetime.timedelta(days=7),
+ )
+
+ job.run()
+
+ self.assertTrue(0 < len(executor.history))
+
+ concurrency_limit_reached_at_least_once = False
+
+ running_tis_total = 0
+
+ for running_tis in executor.history:
+ self.assertLessEqual(len(running_tis), dag.concurrency)
+ running_tis_total += len(running_tis)
+ if len(running_tis) == dag.concurrency:
+ concurrency_limit_reached_at_least_once = True
+
+ self.assertEquals(8, running_tis_total)
+ self.assertTrue(concurrency_limit_reached_at_least_once)
+
Review comment:
Since we have now three similar methods (pool_limit and concurrency_limit
and test_backfill_with_no_pool_limit ) maybe we could actually test if the
right limit has been reached by inspecting the log output. It should be fairly
easy with mocking the log.debug method and checking the log produced as all
three methods produce different method. Then the assertions could be more
accurate (and we could actually check that the right limits are limiting us).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
