syun64 commented on PR #13265:
URL: https://github.com/apache/airflow/pull/13265#issuecomment-1487202191
Hi @kaxil, I'm running an Airflow cluster with v2.5.0, CeleryExecutor and
SQLAlchemy 1.4.4, and I actually ran into the same error noted on this PR.
```
Traceback (most recent call last):
sqlalchemy/engine/base.py", line 1057, in _rollback_impl
self.engine.dialect.do_rollback(self.connection)
sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
psycopg2.DatabaseError: error with status PGRES_TUPLES_OK and no message
from the libpq
```
It seems to have happened when making the call:
airflow/jobs/scheduler_job.py", line 889, in _run_scheduler_loop
num_finished_events = self._process_executor_events(session=session)
On that
[link](https://docs.sqlalchemy.org/en/14/core/pooling.html#using-connection-pools-with-multiprocessing-or-os-fork)
you noted in the PR description, it recommends that we call the dispose
function with `close=False` ([default is
close=True](https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.Engine.dispose))
to ensure that the new process will not touch any of the parent process’
connections and will instead start with new connections.
Was there a reason why we opted to leave the `close` option out of the
function call, or is this something we could add to address more edge cases
where the CeleryExecuter could be going into a bad state when forking on
PostgreSQL connection?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
