[
https://issues.apache.org/jira/browse/AIRFLOW-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Standish updated AIRFLOW-4134:
-------------------------------------
Description:
I am finding with 1.10.2 that I seem to get a warning {{DB connection
invalidated. Reconnecting...}} very frequently.
I to try to diagnose I added logging of the triggering error on line 79 in
airflow/utils/sqlalchemy.py, from which this warning is generated.
Here's the traceback:
{code}
webserver_1 | Traceback (most recent call last):
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/airflow/utils/sqlalchemy.py", line 68,
in ping_connection
webserver_1 | connection.scalar(select([1]))
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 912,
in scalar
webserver_1 | return self.execute(object_, *multiparams, **params).scalar()
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 980,
in execute
webserver_1 | return meth(self, multiparams, params)
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 273,
in _execute_on_connection
webserver_1 | return connection._execute_clauseelement(self, multiparams,
params)
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1099,
in _execute_clauseelement
webserver_1 | distilled_params,
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1240,
in _execute_context
webserver_1 | e, statement, parameters, cursor, context
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1458,
in _handle_dbapi_exception
webserver_1 | util.raise_from_cause(sqlalchemy_exception, exc_info)
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296,
in raise_from_cause
webserver_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276,
in reraise
webserver_1 | raise value.with_traceback(tb)
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236,
in _execute_context
webserver_1 | cursor, statement, parameters, context
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line
536, in do_execute
webserver_1 | cursor.execute(statement, parameters)
webserver_1 | sqlalchemy.exc.OperationalError: (psycopg2.OperationalError)
server closed the connection unexpectedly
webserver_1 | This probably means the server terminated abnormally
webserver_1 | before or while processing the request.
webserver_1 | [SQL: 'SELECT 1'] (Background on this error at:
http://sqlalche.me/e/e3q8)
{code}
It has something to do with the configure_orm function in airflow/settings.py,
because that is the only usage of setup_event_handlers (from
airflow/utils/sqlalchemy.py).
And if I disable connection pooling, then the warning seems to go away.
Beyond that, I am not sure where to go from here. But something must be wrong.
was:
I am finding with 1.10.2 that I seem to get a warning {{DB connection
invalidated. Reconnecting...}} very frequently.
I to try to diagnose I added logging of the triggering error on line 79 in
airflow/utils/sqlalchemy.py, from which this warning is generated.
Here's the traceback:
{code}
webserver_1 | Traceback (most recent call last):
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/airflow/utils/sqlalchemy.py", line 68,
in ping_connection
webserver_1 | connection.scalar(select([1]))
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 912,
in scalar
webserver_1 | return self.execute(object_, *multiparams, **params).scalar()
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 980,
in execute
webserver_1 | return meth(self, multiparams, params)
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 273,
in _execute_on_connection
webserver_1 | return connection._execute_clauseelement(self, multiparams,
params)
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1099,
in _execute_clauseelement
webserver_1 | distilled_params,
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1240,
in _execute_context
webserver_1 | e, statement, parameters, cursor, context
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1458,
in _handle_dbapi_exception
webserver_1 | util.raise_from_cause(sqlalchemy_exception, exc_info)
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296,
in raise_from_cause
webserver_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276,
in reraise
webserver_1 | raise value.with_traceback(tb)
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236,
in _execute_context
webserver_1 | cursor, statement, parameters, context
webserver_1 | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line
536, in do_execute
webserver_1 | cursor.execute(statement, parameters)
webserver_1 | sqlalchemy.exc.OperationalError: (psycopg2.OperationalError)
server closed the connection unexpectedly
webserver_1 | This probably means the server terminated abnormally
webserver_1 | before or while processing the request.
webserver_1 | [SQL: 'SELECT 1'] (Background on this error at:
http://sqlalche.me/e/e3q8)
{code}
It has something to do with the configure_orm function in airflow/settings.py,
because that is the only usage of setup_event_handlers (from
airflow/utils/sqlalchemy.py).
I am not sure where to go from here. But something must be wrong.
> "DB connection invalidated" warning at every zombie check
> ---------------------------------------------------------
>
> Key: AIRFLOW-4134
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4134
> Project: Apache Airflow
> Issue Type: Bug
> Affects Versions: 1.10.2
> Reporter: Daniel Standish
> Priority: Major
>
> I am finding with 1.10.2 that I seem to get a warning {{DB connection
> invalidated. Reconnecting...}} very frequently.
> I to try to diagnose I added logging of the triggering error on line 79 in
> airflow/utils/sqlalchemy.py, from which this warning is generated.
> Here's the traceback:
> {code}
> webserver_1 | Traceback (most recent call last):
> webserver_1 | File
> "/usr/local/lib/python3.6/site-packages/airflow/utils/sqlalchemy.py", line
> 68, in ping_connection
> webserver_1 | connection.scalar(select([1]))
> webserver_1 | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 912,
> in scalar
> webserver_1 | return self.execute(object_, *multiparams,
> **params).scalar()
> webserver_1 | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 980,
> in execute
> webserver_1 | return meth(self, multiparams, params)
> webserver_1 | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line
> 273, in _execute_on_connection
> webserver_1 | return connection._execute_clauseelement(self,
> multiparams, params)
> webserver_1 | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
> 1099, in _execute_clauseelement
> webserver_1 | distilled_params,
> webserver_1 | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
> 1240, in _execute_context
> webserver_1 | e, statement, parameters, cursor, context
> webserver_1 | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
> 1458, in _handle_dbapi_exception
> webserver_1 | util.raise_from_cause(sqlalchemy_exception, exc_info)
> webserver_1 | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296,
> in raise_from_cause
> webserver_1 | reraise(type(exception), exception, tb=exc_tb, cause=cause)
> webserver_1 | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276,
> in reraise
> webserver_1 | raise value.with_traceback(tb)
> webserver_1 | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
> 1236, in _execute_context
> webserver_1 | cursor, statement, parameters, context
> webserver_1 | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line
> 536, in do_execute
> webserver_1 | cursor.execute(statement, parameters)
> webserver_1 | sqlalchemy.exc.OperationalError: (psycopg2.OperationalError)
> server closed the connection unexpectedly
> webserver_1 | This probably means the server terminated abnormally
> webserver_1 | before or while processing the request.
> webserver_1 | [SQL: 'SELECT 1'] (Background on this error at:
> http://sqlalche.me/e/e3q8)
> {code}
> It has something to do with the configure_orm function in
> airflow/settings.py, because that is the only usage of setup_event_handlers
> (from airflow/utils/sqlalchemy.py).
> And if I disable connection pooling, then the warning seems to go away.
> Beyond that, I am not sure where to go from here. But something must be
> wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
