[ https://issues.apache.org/jira/browse/AIRFLOW-622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Imberman closed AIRFLOW-622. ----------------------------------- Resolution: Auto Closed > airflow cannot recover itself after mysql connection was once gone away and > not trigger an error from email alert > ----------------------------------------------------------------------------------------------------------------- > > Key: AIRFLOW-622 > URL: https://issues.apache.org/jira/browse/AIRFLOW-622 > Project: Apache Airflow > Issue Type: Bug > Components: database > Affects Versions: 1.6.2 > Environment: airflow 1.6.1 > Ubuntu 14.04 > Reporter: SHENG JIANG TAO > Priority: Critical > Labels: mysql > > This issue occurs once the connections with airflow mysql gone away, even it > recovered soon, but airflow hang there and print the following errors. > Base on the suggestion of mysql, Session rollback method is needed. > Also not alert mail for this case, and no retry, just silently wrap the log > files. > 2016-11-10 16:04:30,361 [ERROR] file:jobs.py Line:582 - This Session's > transaction has been rolled back by a nested rollback() call. To begin a new > transaction, issue Session.rollback() first. > Traceback (most recent call last): > File > "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/jobs.py", line > 580, in _execute > self.prioritize_queued(executor=executor, dagbag=dagbag) > File > "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/utils.py", > line 122, in wrapper > result = func(*args, **kwargs) > File > "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/jobs.py", line > 482, in prioritize_queued > pools = {p.pool: p for p in session.query(models.Pool).all()} > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2584, in > all > return list(self) > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2732, in > __iter__ > return self._execute_and_instances(context) > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2745, in > _execute_and_instances > close_with_result=True) > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2736, in > _connection_from_session > **kw) > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 905, in > connection > execution_options=execution_options) > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 910, in > _connection_for_bind > engine, execution_options) > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 313, in > _connection_for_bind > self._assert_active() > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 218, in > _assert_active > "This Session's transaction has been rolled back " > InvalidRequestError: This Session's transaction has been rolled back by a > nested rollback() call. To begin a new transaction, issue Session.rollback() > first. -- This message was sent by Atlassian Jira (v8.3.4#803005)