arkadiuszbach opened a new issue, #74614: URL: https://github.com/apache/airflow/issues/74614
On Airflow 3.2.2 (Python 3.12, PostgreSQL), a brief interruption to the metadata database connection caused the triggerer to exit and its pod to restart during `Trigger.clean_unused()`. This issue is also present on `main` at commit [`194e0ea4`](https://github.com/apache/airflow/blob/194e0ea4c9890a7eed1b6df19eb72159f3e44841/airflow-core/src/airflow/models/trigger.py#L233): `Trigger.clean_unused()` retries its UPDATE using `run_with_db_retries()`, which does not roll back the session between attempts. After a connection loss invalidates the transaction, the next attempt reuses that invalid transaction and raises `PendingRollbackError`. ```text 2026-09-14T06:21:13.416868Z [error ] Exception when executing TriggerRunnerSupervisor.run [airflow.jobs.triggerer_job_runner.TriggererJobRunner] loc=triggerer_job_runner.py:221 Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 218, in _execute self.trigger_runner.run() File "/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 614, in run self.clean_unused() File "/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 661, in clean_unused Trigger.clean_unused() File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", line 100, in wrapper return func(*args, session=session, **kwargs) # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/trigger.py", line 230, in clean_unused for attempt in run_with_db_retries(): ^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 438, in __iter__ do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter result = action(retry_state) ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 393, in <lambda> self._add_action_func(lambda rs: rs.outcome.result()) ^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/trigger.py", line 232, in clean_unused session.execute( File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2351, in execute return self._execute_internal( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal result: Result[Any] = compile_state_cls.orm_execute_statement( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/bulk_persistence.py", line 1660, in orm_execute_statement return super().orm_execute_statement( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement result = conn.execute( ^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute return meth( ^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection return connection._execute_clauseelement( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement ret = self._execute_context( ^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context conn = self._revalidate_connection() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection self._invalid_transaction() File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction raise exc.PendingRollbackError( sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b) 2026-09-14T06:21:13.421641Z [info ] Waiting for triggers to clean up [airflow.jobs.triggerer_job_runner.TriggererJobRunner] loc=triggerer_job_runner.py:224 2026-09-14T06:21:13.527625Z [info ] Process exited [supervisor] exit_code=0 loc=supervisor.py:860 pid=29 signal_sent=SIGINT 2026-09-14T06:21:13.527858Z [info ] Exited trigger loop [airflow.jobs.triggerer_job_runner.TriggererJobRunner] loc=triggerer_job_runner.py:229 ``` PR [#70961](https://github.com/apache/airflow/pull/70961) may address this incident because its cleanup retries use a fresh database session for each attempt. --- Drafted-by: Codex (GPT-6.1 Sol); reviewed by @arkadiuszbach before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
