ephraimbuddy commented on code in PR #26205:
URL: https://github.com/apache/airflow/pull/26205#discussion_r968269941
##########
airflow/jobs/backfill_job.py:
##########
@@ -218,6 +218,13 @@ def _update_counters(self, ti_status, session=None):
ti_status.running.pop(reduced_key)
ti_status.to_run[ti.key] = ti
+ elif ti.state == TaskInstanceState.SCHEDULED:
+ # Deferrable task can go from DEFERRED to SCHEDULED,
+ # when that happens, we need to remove it from running
+ # and add it to to_run
+ ti_status.running.pop(ti.key)
+ ti_status.to_run[ti.key] = ti
Review Comment:
As regards the comment, it turns out that kubernetes can also clear queued
runs and they'd get to the scheduled state. This can get the tasks stuck in
scheduled state. While this fix will fix that by having the backfill run it
again, I'm torn between modifying the comment +log here and not clearing
backfill runs in kubernetes. WDYT?
If we don't allow kubernetes to clear the queued runs that have overstayed,
we may run into a case where a backfill task would be stuck in queued.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
