Erik Forsberg created AIRFLOW-1235:
--------------------------------------
Summary: Odd behaviour when all gunicorn workers die
Key: AIRFLOW-1235
URL: https://issues.apache.org/jira/browse/AIRFLOW-1235
Project: Apache Airflow
Issue Type: Bug
Components: webserver
Affects Versions: 1.8.0
Reporter: Erik Forsberg
The webserver has sometimes stopped responding to port 443, and today I found
the issue - I had a misconfigured resolv.conf that made it unable to talk to my
postgresql. This was the root cause, but the way airflow webserver behaved was
a bit odd.
It seems that when all gunicorn workers failed to start, the gunicorn master
shut down. However, the main process (the one that starts gunicorn master) did
not shut down, so there was no way of detecting the failed status of webserver
from e.g. systemd or init script.
Full traceback leading to stale webserver process:
{noformat}
May 21 09:51:57 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] [23794]
[ERROR] Exception in worker process:
May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
1122, in _do_get
May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait,
self._timeout)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
line 145, in get
May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above
exception, another exception occurred:
May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
May 21 09:51:57 airmaster01 airflow[26451]: return fn()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
387, in connect
May 21 09:51:57 airmaster01 airflow[26451]: return
_ConnectionFairy._checkout(self)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
766, in _checkout
May 21 09:51:57 airmaster01 airflow[26451]: fairy =
_ConnectionRecord.checkout(pool)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
516, in checkout
May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
1138, in _do_get
May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value,
exc_tb)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
May 21 09:51:57 airmaster01 airflow[26451]: raise value
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
1135, in _do_get
May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
333, in _create_connection
May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
461, in __init__
May 21 09:51:57 airmaster01 airflow[26451]:
self.__connect(first_connect_check=True)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
651, in __connect
May 21 09:51:57 airmaster01 airflow[26451]: connection =
pool._invoke_creator(self)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
**cparams)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
**cparams)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
connection_factory=connection_factory, **kwasync)
May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could
not translate host name "airstorage01.example.com" to address: Name or service
not known
May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the direct
cause of the following exception:
May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py", line
507, in spawn_worker
May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
line 118, in init_process
May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
line 67, in wsgi
May 21 09:51:57 airmaster01 airflow[26451]: self.callable = self.load()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 65, in load
May 21 09:51:57 airmaster01 airflow[26451]: return self.load_wsgiapp()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 52, in load_wsgiapp
May 21 09:51:57 airmaster01 airflow[26451]: return util.import_app(self.app_uri)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line
366, in import_app
May 21 09:51:57 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
May 21 09:51:57 airmaster01 airflow[26451]: File "<string>", line 1, in <module>
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", line
161, in cached_app
May 21 09:51:57 airmaster01 airflow[26451]: app = create_app(config)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", line
60, in create_app
May 21 09:51:57 airmaster01 airflow[26451]: from airflow.www import views
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 1975, in <module>
May 21 09:51:57 airmaster01 airflow[26451]: class
ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 2057, in ChartModelView
May 21 09:51:57 airmaster01 airflow[26451]: .group_by(models.Connection.conn_id)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2855, in __iter__
May 21 09:51:57 airmaster01 airflow[26451]: return
self._execute_and_instances(context)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2876, in _execute_and_instances
May 21 09:51:57 airmaster01 airflow[26451]: close_with_result=True)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2885, in _get_bind_args
May 21 09:51:57 airmaster01 airflow[26451]: **kw
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2867, in _connection_from_session
May 21 09:51:57 airmaster01 airflow[26451]: conn = self.session.connection(**kw)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 966, in connection
May 21 09:51:57 airmaster01 airflow[26451]: cannot otherwise be identified.
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 971, in _connection_for_bind
May 21 09:51:57 airmaster01 airflow[26451]: closed. This flag only has an
effect if this :class:`.Session` is
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 403, in _connection_for_bind
May 21 09:51:57 airmaster01 airflow[26451]: conn = bind.contextual_connect()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2112, in contextual_connect
May 21 09:51:57 airmaster01 airflow[26451]:
self._wrap_pool_connect(self.pool.connect, None),
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2151, in _wrap_pool_connect
May 21 09:51:57 airmaster01 airflow[26451]: e, dialect, self)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 1465, in _handle_dbapi_exception_noconnection
May 21 09:51:57 airmaster01 airflow[26451]: exc_info
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
May 21 09:51:57 airmaster01 airflow[26451]: reraise(type(exception), exception,
tb=exc_tb, cause=cause)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 186, in reraise
May 21 09:51:57 airmaster01 airflow[26451]: raise value.with_traceback(tb)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
May 21 09:51:57 airmaster01 airflow[26451]: return fn()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
387, in connect
May 21 09:51:57 airmaster01 airflow[26451]: return
_ConnectionFairy._checkout(self)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
766, in _checkout
May 21 09:51:57 airmaster01 airflow[26451]: fairy =
_ConnectionRecord.checkout(pool)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
516, in checkout
May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
1138, in _do_get
May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value,
exc_tb)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
May 21 09:51:57 airmaster01 airflow[26451]: raise value
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
1135, in _do_get
May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
333, in _create_connection
May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
461, in __init__
May 21 09:51:57 airmaster01 airflow[26451]:
self.__connect(first_connect_check=True)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
651, in __connect
May 21 09:51:57 airmaster01 airflow[26451]: connection =
pool._invoke_creator(self)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
**cparams)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
**cparams)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
connection_factory=connection_factory, **kwasync)
May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError:
(psycopg2.OperationalError) could not translate host name
"airstorage01.example.com" to address: Name or service not known
May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
1122, in _do_get
May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait,
self._timeout)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
line 145, in get
May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above
exception, another exception occurred:
May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
May 21 09:51:57 airmaster01 airflow[26451]: return fn()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
387, in connect
May 21 09:51:57 airmaster01 airflow[26451]: return
_ConnectionFairy._checkout(self)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
766, in _checkout
May 21 09:51:57 airmaster01 airflow[26451]: fairy =
_ConnectionRecord.checkout(pool)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
516, in checkout
May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
1138, in _do_get
May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value,
exc_tb)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
May 21 09:51:57 airmaster01 airflow[26451]: raise value
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
1135, in _do_get
May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
333, in _create_connection
May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
461, in __init__
May 21 09:51:57 airmaster01 airflow[26451]:
self.__connect(first_connect_check=True)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
651, in __connect
May 21 09:51:57 airmaster01 airflow[26451]: connection =
pool._invoke_creator(self)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
**cparams)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
**cparams)
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
connection_factory=connection_factory, **kwasync)
May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could
not translate host name "airstorage01.example.com" to address: Name or service
not known
May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the direct
cause of the following exception:
May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py", line
507, in spawn_worker
May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
line 118, in init_process
May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
May 21 09:51:57 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
line 67, in wsgi
May 21 09:51:58 airmaster01 airflow[26451]: self.callable = self.load()
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 65, in load
May 21 09:51:58 airmaster01 airflow[26451]: return self.load_wsgiapp()
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
line 52, in load_wsgiapp
May 21 09:51:58 airmaster01 airflow[26451]: return util.import_app(self.app_uri)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line
366, in import_app
May 21 09:51:58 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
May 21 09:51:58 airmaster01 airflow[26451]: File "<string>", line 1, in <module>
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", line
161, in cached_app
May 21 09:51:58 airmaster01 airflow[26451]: app = create_app(config)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", line
60, in create_app
May 21 09:51:58 airmaster01 airflow[26451]: from airflow.www import views
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 1975, in <module>
May 21 09:51:58 airmaster01 airflow[26451]: class
ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
line 2057, in ChartModelView
May 21 09:51:58 airmaster01 airflow[26451]: .group_by(models.Connection.conn_id)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2855, in __iter__
May 21 09:51:58 airmaster01 airflow[26451]: return
self._execute_and_instances(context)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2876, in _execute_and_instances
May 21 09:51:58 airmaster01 airflow[26451]: close_with_result=True)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2885, in _get_bind_args
May 21 09:51:58 airmaster01 airflow[26451]: **kw
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 2867, in _connection_from_session
May 21 09:51:58 airmaster01 airflow[26451]: conn = self.session.connection(**kw)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 966, in connection
May 21 09:51:58 airmaster01 airflow[26451]: cannot otherwise be identified.
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 971, in _connection_for_bind
May 21 09:51:58 airmaster01 airflow[26451]: closed. This flag only has an
effect if this :class:`.Session` is
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 403, in _connection_for_bind
May 21 09:51:58 airmaster01 airflow[26451]: conn = bind.contextual_connect()
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2112, in contextual_connect
May 21 09:51:58 airmaster01 airflow[26451]:
self._wrap_pool_connect(self.pool.connect, None),
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2151, in _wrap_pool_connect
May 21 09:51:58 airmaster01 airflow[26451]: e, dialect, self)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 1465, in _handle_dbapi_exception_noconnection
May 21 09:51:58 airmaster01 airflow[26451]: exc_info
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
May 21 09:51:58 airmaster01 airflow[26451]: reraise(type(exception), exception,
tb=exc_tb, cause=cause)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 186, in reraise
May 21 09:51:58 airmaster01 airflow[26451]: raise value.with_traceback(tb)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
May 21 09:51:58 airmaster01 airflow[26451]: return fn()
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
387, in connect
May 21 09:51:58 airmaster01 airflow[26451]: return
_ConnectionFairy._checkout(self)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
766, in _checkout
May 21 09:51:58 airmaster01 airflow[26451]: fairy =
_ConnectionRecord.checkout(pool)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
516, in checkout
May 21 09:51:58 airmaster01 airflow[26451]: rec = pool._do_get()
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
1138, in _do_get
May 21 09:51:58 airmaster01 airflow[26451]: self._dec_overflow()
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
May 21 09:51:58 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value,
exc_tb)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
May 21 09:51:58 airmaster01 airflow[26451]: raise value
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
1135, in _do_get
May 21 09:51:58 airmaster01 airflow[26451]: return self._create_connection()
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
333, in _create_connection
May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionRecord(self)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
461, in __init__
May 21 09:51:58 airmaster01 airflow[26451]:
self.__connect(first_connect_check=True)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line
651, in __connect
May 21 09:51:58 airmaster01 airflow[26451]: connection =
pool._invoke_creator(self)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
May 21 09:51:58 airmaster01 airflow[26451]: return dialect.connect(*cargs,
**cparams)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
May 21 09:51:58 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
**cparams)
May 21 09:51:58 airmaster01 airflow[26451]: File
"/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
line 130, in connect
May 21 09:51:58 airmaster01 airflow[26451]: conn = _connect(dsn,
connection_factory=connection_factory, **kwasync)
May 21 09:51:58 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError:
(psycopg2.OperationalError) could not translate host name
"airstorage01.example.com" to address: Name or service not known
May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] [23794]
[INFO] Worker exiting (pid: 23794)
May 21 09:51:58 airmaster01 airflow[26451]:
/opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache
instead.
May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
ExtDeprecationWarning
May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23561]
[INFO] Worker exiting (pid: 23561)
May 21 09:51:58 airmaster01 airflow[26451]:
/opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache
instead.
May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
ExtDeprecationWarning
May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23759]
[INFO] Worker exiting (pid: 23759)
May 21 09:51:58 airmaster01 airflow[26451]:
/opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache
instead.
May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
ExtDeprecationWarning
May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23620]
[INFO] Worker exiting (pid: 23620)
May 21 09:51:58 airmaster01 airflow[26451]:
/opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache
instead.
May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
ExtDeprecationWarning
May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23689]
[INFO] Worker exiting (pid: 23689)
May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] [26464]
[INFO] Shutting down: Master
May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] [26464]
[INFO] Reason: Worker failed to boot.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
