[ 
https://issues.apache.org/jira/browse/AIRFLOW-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397679#comment-16397679
 ] 

John Arnold commented on AIRFLOW-1235:
--------------------------------------

Does anyone know how to fix this so that either gunicorn gets restarted, or the 
main process dies?

> Odd behaviour when all gunicorn workers die
> -------------------------------------------
>
>                 Key: AIRFLOW-1235
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1235
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: webserver
>    Affects Versions: 1.8.0
>            Reporter: Erik Forsberg
>            Assignee: Kengo Seki
>            Priority: Major
>
> The webserver has sometimes stopped responding to port 443, and today I found 
> the issue - I had a misconfigured resolv.conf that made it unable to talk to 
> my postgresql. This was the root cause, but the way airflow webserver behaved 
> was a bit odd.
> It seems that when all gunicorn workers failed to start, the gunicorn master 
> shut down. However, the main process (the one that starts gunicorn master) 
> did not shut down, so there was no way of detecting the failed status of 
> webserver from e.g. systemd or init script.
> Full traceback leading to stale webserver process:
> {noformat}
> May 21 09:51:57 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] 
> [23794] [ERROR] Exception in worker process:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait, 
> self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
>  line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above 
> exception, another exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
>  line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return 
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy = 
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
>  line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, 
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
>  line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]: 
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection = 
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
>  line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, 
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
>  line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, 
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", 
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, 
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could 
> not translate host name "airstorage01.example.com" to address: Name or 
> service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the 
> direct cause of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py", 
> line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
>  line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py", 
> line 67, in wsgi
> May 21 09:51:57 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
>  line 65, in load
> May 21 09:51:57 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
>  line 52, in load_wsgiapp
> May 21 09:51:57 airmaster01 airflow[26451]: return 
> util.import_app(self.app_uri)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line 
> 366, in import_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:57 airmaster01 airflow[26451]: File "<string>", line 1, in 
> <module>
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", 
> line 161, in cached_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", 
> line 60, in create_app
> May 21 09:51:57 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", 
> line 1975, in <module>
> May 21 09:51:57 airmaster01 airflow[26451]: class 
> ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", 
> line 2057, in ChartModelView
> May 21 09:51:57 airmaster01 airflow[26451]: 
> .group_by(models.Connection.conn_id)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
>  line 2855, in __iter__
> May 21 09:51:57 airmaster01 airflow[26451]: return 
> self._execute_and_instances(context)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
>  line 2876, in _execute_and_instances
> May 21 09:51:57 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
>  line 2885, in _get_bind_args
> May 21 09:51:57 airmaster01 airflow[26451]: **kw
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
>  line 2867, in _connection_from_session
> May 21 09:51:57 airmaster01 airflow[26451]: conn = 
> self.session.connection(**kw)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
>  line 966, in connection
> May 21 09:51:57 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
>  line 971, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: closed.  This flag only has an 
> effect if this :class:`.Session` is
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
>  line 403, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
>  line 2112, in contextual_connect
> May 21 09:51:57 airmaster01 airflow[26451]: 
> self._wrap_pool_connect(self.pool.connect, None),
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
>  line 2151, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
>  line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:57 airmaster01 airflow[26451]: exc_info
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
>  line 203, in raise_from_cause
> May 21 09:51:57 airmaster01 airflow[26451]: reraise(type(exception), 
> exception, tb=exc_tb, cause=cause)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
>  line 186, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
>  line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return 
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy = 
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
>  line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, 
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
>  line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]: 
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection = 
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
>  line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, 
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
>  line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, 
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", 
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, 
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError: 
> (psycopg2.OperationalError) could not translate host name 
> "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait, 
> self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
>  line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above 
> exception, another exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
>  line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return 
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy = 
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
>  line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, 
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
>  line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]: 
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection = 
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
>  line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, 
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
>  line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, 
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", 
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, 
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could 
> not translate host name "airstorage01.example.com" to address: Name or 
> service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the 
> direct cause of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py", 
> line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
>  line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py", 
> line 67, in wsgi
> May 21 09:51:58 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
>  line 65, in load
> May 21 09:51:58 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
>  line 52, in load_wsgiapp
> May 21 09:51:58 airmaster01 airflow[26451]: return 
> util.import_app(self.app_uri)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line 
> 366, in import_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:58 airmaster01 airflow[26451]: File "<string>", line 1, in 
> <module>
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", 
> line 161, in cached_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", 
> line 60, in create_app
> May 21 09:51:58 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", 
> line 1975, in <module>
> May 21 09:51:58 airmaster01 airflow[26451]: class 
> ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", 
> line 2057, in ChartModelView
> May 21 09:51:58 airmaster01 airflow[26451]: 
> .group_by(models.Connection.conn_id)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
>  line 2855, in __iter__
> May 21 09:51:58 airmaster01 airflow[26451]: return 
> self._execute_and_instances(context)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
>  line 2876, in _execute_and_instances
> May 21 09:51:58 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
>  line 2885, in _get_bind_args
> May 21 09:51:58 airmaster01 airflow[26451]: **kw
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
>  line 2867, in _connection_from_session
> May 21 09:51:58 airmaster01 airflow[26451]: conn = 
> self.session.connection(**kw)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
>  line 966, in connection
> May 21 09:51:58 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
>  line 971, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: closed.  This flag only has an 
> effect if this :class:`.Session` is
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
>  line 403, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
>  line 2112, in contextual_connect
> May 21 09:51:58 airmaster01 airflow[26451]: 
> self._wrap_pool_connect(self.pool.connect, None),
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
>  line 2151, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
>  line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:58 airmaster01 airflow[26451]: exc_info
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
>  line 203, in raise_from_cause
> May 21 09:51:58 airmaster01 airflow[26451]: reraise(type(exception), 
> exception, tb=exc_tb, cause=cause)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
>  line 186, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
>  line 2147, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: return fn()
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 387, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return 
> _ConnectionFairy._checkout(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 766, in _checkout
> May 21 09:51:58 airmaster01 airflow[26451]: fairy = 
> _ConnectionRecord.checkout(pool)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 516, in checkout
> May 21 09:51:58 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 1138, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
>  line 66, in __exit__
> May 21 09:51:58 airmaster01 airflow[26451]: compat.reraise(exc_type, 
> exc_value, exc_tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
>  line 187, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 1135, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 333, in _create_connection
> May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 461, in __init__
> May 21 09:51:58 airmaster01 airflow[26451]: 
> self.__connect(first_connect_check=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", 
> line 651, in __connect
> May 21 09:51:58 airmaster01 airflow[26451]: connection = 
> pool._invoke_creator(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
>  line 105, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return dialect.connect(*cargs, 
> **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
>  line 393, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, 
> **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File 
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", 
> line 130, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: conn = _connect(dsn, 
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:58 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError: 
> (psycopg2.OperationalError) could not translate host name 
> "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] 
> [23794] [INFO] Worker exiting (pid: 23794)
> May 21 09:51:58 airmaster01 airflow[26451]: 
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: 
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use 
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), 
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] 
> [23561] [INFO] Worker exiting (pid: 23561)
> May 21 09:51:58 airmaster01 airflow[26451]: 
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: 
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use 
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), 
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] 
> [23759] [INFO] Worker exiting (pid: 23759)
> May 21 09:51:58 airmaster01 airflow[26451]: 
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: 
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use 
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), 
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] 
> [23620] [INFO] Worker exiting (pid: 23620)
> May 21 09:51:58 airmaster01 airflow[26451]: 
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: 
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use 
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), 
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] 
> [23689] [INFO] Worker exiting (pid: 23689)
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] 
> [26464] [INFO] Shutting down: Master
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] 
> [26464] [INFO] Reason: Worker failed to boot.
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to