[
https://issues.apache.org/jira/browse/AIRFLOW-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298626#comment-16298626
]
Adam Hodges commented on AIRFLOW-1235:
--------------------------------------
I am also seeing this issue when we have connectivity issues to our Postgres
database. All the workers die and then the gunicorn master dies, but the
airflow process does not die. If the airflow process died, I could at least
detect it and restart the airflow webserver.
My logs:
{noformat}
[2017-12-20 00:56:26 +0000] [844] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507,
in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line
118, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67,
in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line
65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line
52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 366, in
import_app
app = eval(obj, mod.__dict__)
File "<string>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 161,
in cached_app
app = create_app(config)
File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 60, in
create_app
from airflow.www import views
File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line
1977, in <module>
class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line
2059, in ChartModelView
.group_by(models.Connection.conn_id)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
2855, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
2876, in _execute_and_instances
close_with_result=True)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
2885, in _get_bind_args
**kw
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
2867, in _connection_from_session
conn = self.session.connection(**kw)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line
998, in connection
execution_options=execution_options)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line
1003, in _connection_for_bind
engine, execution_options)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line
403, in _connection_for_bind
conn = bind.contextual_connect()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
2112, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
2151, in _wrap_pool_connect
e, dialect, self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
1465, in _handle_dbapi_exception_noconnection
exc_info
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line
203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
2147, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 387,
in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 766,
in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 516,
in checkout
rec = pool._do_get()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1138,
in _do_get
self._dec_overflow()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1135,
in _do_get
return self._create_connection()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 333,
in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 461,
in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 651,
in __connect
connection = pool._invoke_creator(self)
File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line
105, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
line 393, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py", line 130,
in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
OperationalError: (psycopg2.OperationalError) could not translate host name
"postgres" to address: Name or service not known
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507,
in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line
118, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67,
in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line
65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line
52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 366, in
import_app
app = eval(obj, mod.__dict__)
File "<string>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 161,
in cached_app
app = create_app(config)
File "/usr/local/lib/python2.7/dist-packages/airflow/www/app.py", line 60, in
create_app
from airflow.www import views
File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line
1977, in <module>
class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line
2059, in ChartModelView
.group_by(models.Connection.conn_id)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
2855, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
2876, in _execute_and_instances
close_with_result=True)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
2885, in _get_bind_args
**kw
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
2867, in _connection_from_session
conn = self.session.connection(**kw)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line
998, in connection
execution_options=execution_options)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line
1003, in _connection_for_bind
engine, execution_options)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line
403, in _connection_for_bind
conn = bind.contextual_connect()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
2112, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
2151, in _wrap_pool_connect
e, dialect, self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
1465, in _handle_dbapi_exception_noconnection
exc_info
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line
203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
2147, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 387,
in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 766,
in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 516,
in checkout
rec = pool._do_get()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1138,
in _do_get
self._dec_overflow()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1135,
in _do_get
return self._create_connection()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 333,
in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 461,
in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 651,
in __connect
connection = pool._invoke_creator(self)
File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line
105, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
line 393, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py", line 130,
in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
OperationalError: (psycopg2.OperationalError) could not translate host name
"postgres" to address: Name or service not known
[2017-12-20 00:56:26 +0000] [844] [INFO] Worker exiting (pid: 844)
[2017-12-20 00:56:26 +0000] [842] [INFO] Worker exiting (pid: 842)
[2017-12-20 00:56:26 +0000] [840] [INFO] Worker exiting (pid: 840)
[2017-12-20 00:56:26 +0000] [841] [INFO] Worker exiting (pid: 841)
[2017-12-20 00:56:26 +0000] [843] [INFO] Worker exiting (pid: 843)
[2017-12-20 00:56:26 +0000] [41] [INFO] Shutting down: Master
[2017-12-20 00:56:26 +0000] [41] [INFO] Reason: Worker failed to boot.
[2017-12-20 00:56:26,940] [1] {cli.py:723} ERROR - [0 / 0] some workers seem to
have died and gunicorndid not restart them as expected
{noformat}
Either the master gunicorn process should be restarted if it dies, or the
airflow webserver should die along with the master gunicorn process.
> Odd behaviour when all gunicorn workers die
> -------------------------------------------
>
> Key: AIRFLOW-1235
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1235
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
> Affects Versions: 1.8.0
> Reporter: Erik Forsberg
> Assignee: Kengo Seki
>
> The webserver has sometimes stopped responding to port 443, and today I found
> the issue - I had a misconfigured resolv.conf that made it unable to talk to
> my postgresql. This was the root cause, but the way airflow webserver behaved
> was a bit odd.
> It seems that when all gunicorn workers failed to start, the gunicorn master
> shut down. However, the main process (the one that starts gunicorn master)
> did not shut down, so there was no way of detecting the failed status of
> webserver from e.g. systemd or init script.
> Full traceback leading to stale webserver process:
> {noformat}
> May 21 09:51:57 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000]
> [23794] [ERROR] Exception in worker process:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait,
> self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
> line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above
> exception, another exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could
> not translate host name "airstorage01.example.com" to address: Name or
> service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the
> direct cause of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
> line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
> line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
> line 67, in wsgi
> May 21 09:51:57 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 65, in load
> May 21 09:51:57 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 52, in load_wsgiapp
> May 21 09:51:57 airmaster01 airflow[26451]: return
> util.import_app(self.app_uri)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line
> 366, in import_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:57 airmaster01 airflow[26451]: File "<string>", line 1, in
> <module>
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 161, in cached_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 60, in create_app
> May 21 09:51:57 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 1975, in <module>
> May 21 09:51:57 airmaster01 airflow[26451]: class
> ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 2057, in ChartModelView
> May 21 09:51:57 airmaster01 airflow[26451]:
> .group_by(models.Connection.conn_id)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2855, in __iter__
> May 21 09:51:57 airmaster01 airflow[26451]: return
> self._execute_and_instances(context)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2876, in _execute_and_instances
> May 21 09:51:57 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2885, in _get_bind_args
> May 21 09:51:57 airmaster01 airflow[26451]: **kw
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2867, in _connection_from_session
> May 21 09:51:57 airmaster01 airflow[26451]: conn =
> self.session.connection(**kw)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 966, in connection
> May 21 09:51:57 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 971, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: closed. This flag only has an
> effect if this :class:`.Session` is
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 403, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2112, in contextual_connect
> May 21 09:51:57 airmaster01 airflow[26451]:
> self._wrap_pool_connect(self.pool.connect, None),
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2151, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:57 airmaster01 airflow[26451]: exc_info
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 203, in raise_from_cause
> May 21 09:51:57 airmaster01 airflow[26451]: reraise(type(exception),
> exception, tb=exc_tb, cause=cause)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 186, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError:
> (psycopg2.OperationalError) could not translate host name
> "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait,
> self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
> line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above
> exception, another exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could
> not translate host name "airstorage01.example.com" to address: Name or
> service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the
> direct cause of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
> line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
> line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
> line 67, in wsgi
> May 21 09:51:58 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 65, in load
> May 21 09:51:58 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 52, in load_wsgiapp
> May 21 09:51:58 airmaster01 airflow[26451]: return
> util.import_app(self.app_uri)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line
> 366, in import_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:58 airmaster01 airflow[26451]: File "<string>", line 1, in
> <module>
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 161, in cached_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 60, in create_app
> May 21 09:51:58 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 1975, in <module>
> May 21 09:51:58 airmaster01 airflow[26451]: class
> ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 2057, in ChartModelView
> May 21 09:51:58 airmaster01 airflow[26451]:
> .group_by(models.Connection.conn_id)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2855, in __iter__
> May 21 09:51:58 airmaster01 airflow[26451]: return
> self._execute_and_instances(context)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2876, in _execute_and_instances
> May 21 09:51:58 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2885, in _get_bind_args
> May 21 09:51:58 airmaster01 airflow[26451]: **kw
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2867, in _connection_from_session
> May 21 09:51:58 airmaster01 airflow[26451]: conn =
> self.session.connection(**kw)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 966, in connection
> May 21 09:51:58 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 971, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: closed. This flag only has an
> effect if this :class:`.Session` is
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 403, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2112, in contextual_connect
> May 21 09:51:58 airmaster01 airflow[26451]:
> self._wrap_pool_connect(self.pool.connect, None),
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2151, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:58 airmaster01 airflow[26451]: exc_info
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 203, in raise_from_cause
> May 21 09:51:58 airmaster01 airflow[26451]: reraise(type(exception),
> exception, tb=exc_tb, cause=cause)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 186, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: return fn()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:58 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:58 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:58 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:58 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:58 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:58 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError:
> (psycopg2.OperationalError) could not translate host name
> "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000]
> [23794] [INFO] Worker exiting (pid: 23794)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23561] [INFO] Worker exiting (pid: 23561)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23759] [INFO] Worker exiting (pid: 23759)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23620] [INFO] Worker exiting (pid: 23620)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23689] [INFO] Worker exiting (pid: 23689)
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000]
> [26464] [INFO] Shutting down: Master
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000]
> [26464] [INFO] Reason: Worker failed to boot.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
