[
https://issues.apache.org/jira/browse/AIRFLOW-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kengo Seki reassigned AIRFLOW-1235:
-----------------------------------
Assignee: Kengo Seki
> Odd behaviour when all gunicorn workers die
> -------------------------------------------
>
> Key: AIRFLOW-1235
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1235
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
> Affects Versions: 1.8.0
> Reporter: Erik Forsberg
> Assignee: Kengo Seki
>
> The webserver has sometimes stopped responding to port 443, and today I found
> the issue - I had a misconfigured resolv.conf that made it unable to talk to
> my postgresql. This was the root cause, but the way airflow webserver behaved
> was a bit odd.
> It seems that when all gunicorn workers failed to start, the gunicorn master
> shut down. However, the main process (the one that starts gunicorn master)
> did not shut down, so there was no way of detecting the failed status of
> webserver from e.g. systemd or init script.
> Full traceback leading to stale webserver process:
> {noformat}
> May 21 09:51:57 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000]
> [23794] [ERROR] Exception in worker process:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait,
> self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
> line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above
> exception, another exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could
> not translate host name "airstorage01.example.com" to address: Name or
> service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the
> direct cause of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
> line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
> line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
> line 67, in wsgi
> May 21 09:51:57 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 65, in load
> May 21 09:51:57 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 52, in load_wsgiapp
> May 21 09:51:57 airmaster01 airflow[26451]: return
> util.import_app(self.app_uri)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line
> 366, in import_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:57 airmaster01 airflow[26451]: File "<string>", line 1, in
> <module>
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 161, in cached_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 60, in create_app
> May 21 09:51:57 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 1975, in <module>
> May 21 09:51:57 airmaster01 airflow[26451]: class
> ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 2057, in ChartModelView
> May 21 09:51:57 airmaster01 airflow[26451]:
> .group_by(models.Connection.conn_id)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2855, in __iter__
> May 21 09:51:57 airmaster01 airflow[26451]: return
> self._execute_and_instances(context)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2876, in _execute_and_instances
> May 21 09:51:57 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2885, in _get_bind_args
> May 21 09:51:57 airmaster01 airflow[26451]: **kw
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2867, in _connection_from_session
> May 21 09:51:57 airmaster01 airflow[26451]: conn =
> self.session.connection(**kw)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 966, in connection
> May 21 09:51:57 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 971, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: closed. This flag only has an
> effect if this :class:`.Session` is
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 403, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2112, in contextual_connect
> May 21 09:51:57 airmaster01 airflow[26451]:
> self._wrap_pool_connect(self.pool.connect, None),
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2151, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:57 airmaster01 airflow[26451]: exc_info
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 203, in raise_from_cause
> May 21 09:51:57 airmaster01 airflow[26451]: reraise(type(exception),
> exception, tb=exc_tb, cause=cause)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 186, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError:
> (psycopg2.OperationalError) could not translate host name
> "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait,
> self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
> line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above
> exception, another exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could
> not translate host name "airstorage01.example.com" to address: Name or
> service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the
> direct cause of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
> line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
> line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
> line 67, in wsgi
> May 21 09:51:58 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 65, in load
> May 21 09:51:58 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 52, in load_wsgiapp
> May 21 09:51:58 airmaster01 airflow[26451]: return
> util.import_app(self.app_uri)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line
> 366, in import_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:58 airmaster01 airflow[26451]: File "<string>", line 1, in
> <module>
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 161, in cached_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 60, in create_app
> May 21 09:51:58 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 1975, in <module>
> May 21 09:51:58 airmaster01 airflow[26451]: class
> ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 2057, in ChartModelView
> May 21 09:51:58 airmaster01 airflow[26451]:
> .group_by(models.Connection.conn_id)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2855, in __iter__
> May 21 09:51:58 airmaster01 airflow[26451]: return
> self._execute_and_instances(context)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2876, in _execute_and_instances
> May 21 09:51:58 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2885, in _get_bind_args
> May 21 09:51:58 airmaster01 airflow[26451]: **kw
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2867, in _connection_from_session
> May 21 09:51:58 airmaster01 airflow[26451]: conn =
> self.session.connection(**kw)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 966, in connection
> May 21 09:51:58 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 971, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: closed. This flag only has an
> effect if this :class:`.Session` is
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 403, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2112, in contextual_connect
> May 21 09:51:58 airmaster01 airflow[26451]:
> self._wrap_pool_connect(self.pool.connect, None),
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2151, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:58 airmaster01 airflow[26451]: exc_info
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 203, in raise_from_cause
> May 21 09:51:58 airmaster01 airflow[26451]: reraise(type(exception),
> exception, tb=exc_tb, cause=cause)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 186, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: return fn()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:58 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:58 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:58 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:58 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:58 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:58 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError:
> (psycopg2.OperationalError) could not translate host name
> "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000]
> [23794] [INFO] Worker exiting (pid: 23794)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23561] [INFO] Worker exiting (pid: 23561)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23759] [INFO] Worker exiting (pid: 23759)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23620] [INFO] Worker exiting (pid: 23620)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23689] [INFO] Worker exiting (pid: 23689)
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000]
> [26464] [INFO] Shutting down: Master
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000]
> [26464] [INFO] Reason: Worker failed to boot.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
