This is an automated email from the ASF dual-hosted git repository.
dstandish pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new c9e190fb254 Add todo re exception handling in dagrun create loop
(#59121)
c9e190fb254 is described below
commit c9e190fb25485c50f7d2b9042ede735413cd1935
Author: Daniel Standish <[email protected]>
AuthorDate: Thu Dec 11 08:56:19 2025 -0800
Add todo re exception handling in dagrun create loop (#59121)
In SchedulerJobRunner, in _create_dag_runs, we do except Exception: ...
continue but this does not work when the exception is the result of a db error;
in that case you need to roll the transaction back.
To fix this we need either smaller transactions (e.g. one per dag run
create attempt) or savepoints. either way it's a bit of a refactor.
Issue: https://github.com/apache/airflow/issues/59120
---
airflow-core/src/airflow/jobs/scheduler_job_runner.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/airflow-core/src/airflow/jobs/scheduler_job_runner.py
b/airflow-core/src/airflow/jobs/scheduler_job_runner.py
index 5d8df8e970e..ceff2ff5ca6 100644
--- a/airflow-core/src/airflow/jobs/scheduler_job_runner.py
+++ b/airflow-core/src/airflow/jobs/scheduler_job_runner.py
@@ -1831,6 +1831,10 @@ class SchedulerJobRunner(BaseJobRunner, LoggingMixin):
# and continue to the next serdag.
except Exception:
self.log.exception("Failed creating DagRun for %s",
serdag.dag_id)
+ # todo: continuing here does not work because session
needs rollback
+ # but you need either to make smaller transactions and
commit after every dag run
+ # or to use savepoints.
+ # https://github.com/apache/airflow/issues/59120
continue
if self._should_update_dag_next_dagruns(
serdag,
