[
https://issues.apache.org/jira/browse/AIRFLOW-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405547#comment-16405547
]
John Arnold commented on AIRFLOW-1235:
--------------------------------------
For reference, here's what it looks linke in my logs too...
```
[2018-03-19 03:21:31,442] \{configuration.py:211} WARNING - section/key
[celery/ssl_active] not found in config
[2018-03-19 03:21:31,442] \{default_celery.py:44} WARNING - Celery Executor
will run without SSL
[2018-03-19 03:21:33,299] \{config.py:26} INFO - Missing required environment
vars. Attempting to load from /etc/celery/conf.d/celery
[2018-03-19 03:21:33,352] \{__init__.py:45} INFO - Using executor CeleryExecutor
[2018-03-19 03:21:33,724] \{models.py:196} INFO - Filling up the DagBag from
/var/lib/airflow/venv/lib/python3.6/site-packages/net_task_contrib/airflow/dags
[2018-03-19 03:21:36 +0000] [125724] [INFO] Handling signal: ttou
[2018-03-19 03:21:36 +0000] [55375] [INFO] Worker exiting (pid: 55375)
[2018-03-19 03:22:07 +0000] [125724] [INFO] Handling signal: ttin
[2018-03-19 03:22:07 +0000] [59230] [INFO] Booting worker with pid: 59230
[2018-03-19 03:22:07,621] \{configuration.py:211} WARNING - section/key
[celery/ssl_active] not found in config
[2018-03-19 03:22:07,621] \{default_celery.py:44} WARNING - Celery Executor
will run without SSL
[2018-03-19 03:22:08,604] \{config.py:26} INFO - Missing required environment
vars. Attempting to load from /etc/celery/conf.d/celery
[2018-03-19 03:22:08,648] \{__init__.py:45} INFO - Using executor CeleryExecutor
[2018-03-19 03:22:08,979] \{models.py:196} INFO - Filling up the DagBag from
/var/lib/airflow/venv/lib/python3.6/site-packages/net_task_contrib/airflow/dags
[2018-03-19 03:22:09 +0000] [59230] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 1124, in _do_get
return self._pool.get(wait, self._timeout)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/queue.py",
line 145, in get
raise Empty
sqlalchemy.util.queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
return fn()
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 387, in connect
return _ConnectionFairy._checkout(self)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 768, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 516, in checkout
rec = pool._do_get()
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 1140, in _do_get
self._dec_overflow()
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
raise value
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 1137, in _do_get
return self._create_connection()
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 333, in _create_connection
return _ConnectionRecord(self)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 461, in __init__
self.__connect(first_connect_check=True)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 651, in __connect
connection = pool._invoke_creator(self)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
return dialect.connect(*cargs, **cparams)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/psycopg2/__init__.py",
line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: SSL SYSCALL error: EOF detected
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/arbiter.py",
line 578, in spawn_worker
worker.init_process()
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/workers/base.py",
line 126, in init_process
self.load_wsgi()
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/workers/base.py",
line 135, in load_wsgi
self.wsgi = self.app.wsgi()
File "/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/app/base.py",
line 67, in wsgi
self.callable = self.load()
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py",
line 65, in load
return self.load_wsgiapp()
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py",
line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/gunicorn/util.py",
line 364, in import_app
app = eval(obj, vars(mod))
File "<string>", line 1, in <module>
File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/www/app.py",
line 177, in cached_app
app = create_app(config, testing)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/www/app.py",
line 63, in create_app
from airflow.www import views
File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/www/views.py",
line 2099, in <module>
class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/www/views.py",
line 2190, in ChartModelView
'conn_id': _connection_ids()
File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/utils/db.py",
line 69, in wrapper
return func(*args, **kwargs)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/airflow/www/views.py",
line 2094, in _connection_ids
.group_by(models.Connection.conn_id)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py",
line 2855, in __iter__
return self._execute_and_instances(context)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py",
line 2876, in _execute_and_instances
close_with_result=True)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py",
line 2885, in _get_bind_args
**kw
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py",
line 2867, in _connection_from_session
conn = self.session.connection(**kw)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py",
line 1019, in connection
execution_options=execution_options)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py",
line 1024, in _connection_for_bind
engine, execution_options)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py",
line 409, in _connection_for_bind
conn = bind.contextual_connect()
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 2112, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 2151, in _wrap_pool_connect
e, dialect, self)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1465, in _handle_dbapi_exception_noconnection
exc_info
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py",
line 186, in reraise
raise value.with_traceback(tb)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 2147, in _wrap_pool_connect
return fn()
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 387, in connect
return _ConnectionFairy._checkout(self)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 768, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 516, in checkout
rec = pool._do_get()
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 1140, in _do_get
self._dec_overflow()
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py",
line 187, in reraise
raise value
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 1137, in _do_get
return self._create_connection()
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 333, in _create_connection
return _ConnectionRecord(self)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 461, in __init__
self.__connect(first_connect_check=True)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py",
line 651, in __connect
connection = pool._invoke_creator(self)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
return dialect.connect(*cargs, **cparams)
File
"/var/lib/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
line 393, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/var/lib/airflow/venv/lib/python3.6/site-packages/psycopg2/__init__.py",
line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error:
EOF detected
[2018-03-19 03:22:09 +0000] [59230] [INFO] Worker exiting (pid: 59230)
[2018-03-19 03:22:09 +0000] [57686] [INFO] Worker exiting (pid: 57686)
[2018-03-19 03:22:09 +0000] [58454] [INFO] Worker exiting (pid: 58454)
[2018-03-19 03:22:09 +0000] [56148] [INFO] Worker exiting (pid: 56148)
[2018-03-19 03:22:09 +0000] [56923] [INFO] Worker exiting (pid: 56923)
[2018-03-19 03:22:10 +0000] [125724] [INFO] Shutting down: Master
[2018-03-19 03:22:10 +0000] [125724] [INFO] Reason: Worker failed to boot.
[2018-03-19 03:22:10,322] \{cli.py:686} ERROR - [0 / 0] some workers seem to
have died and gunicorndid not restart them as expected
> Odd behaviour when all gunicorn workers die
> -------------------------------------------
>
> Key: AIRFLOW-1235
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1235
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
> Affects Versions: 1.8.0
> Reporter: Erik Forsberg
> Assignee: Kengo Seki
> Priority: Major
>
> The webserver has sometimes stopped responding to port 443, and today I found
> the issue - I had a misconfigured resolv.conf that made it unable to talk to
> my postgresql. This was the root cause, but the way airflow webserver behaved
> was a bit odd.
> It seems that when all gunicorn workers failed to start, the gunicorn master
> shut down. However, the main process (the one that starts gunicorn master)
> did not shut down, so there was no way of detecting the failed status of
> webserver from e.g. systemd or init script.
> Full traceback leading to stale webserver process:
> {noformat}
> May 21 09:51:57 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000]
> [23794] [ERROR] Exception in worker process:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait,
> self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
> line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above
> exception, another exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could
> not translate host name "airstorage01.example.com" to address: Name or
> service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the
> direct cause of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
> line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
> line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
> line 67, in wsgi
> May 21 09:51:57 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 65, in load
> May 21 09:51:57 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 52, in load_wsgiapp
> May 21 09:51:57 airmaster01 airflow[26451]: return
> util.import_app(self.app_uri)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line
> 366, in import_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:57 airmaster01 airflow[26451]: File "<string>", line 1, in
> <module>
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 161, in cached_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 60, in create_app
> May 21 09:51:57 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 1975, in <module>
> May 21 09:51:57 airmaster01 airflow[26451]: class
> ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 2057, in ChartModelView
> May 21 09:51:57 airmaster01 airflow[26451]:
> .group_by(models.Connection.conn_id)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2855, in __iter__
> May 21 09:51:57 airmaster01 airflow[26451]: return
> self._execute_and_instances(context)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2876, in _execute_and_instances
> May 21 09:51:57 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2885, in _get_bind_args
> May 21 09:51:57 airmaster01 airflow[26451]: **kw
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2867, in _connection_from_session
> May 21 09:51:57 airmaster01 airflow[26451]: conn =
> self.session.connection(**kw)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 966, in connection
> May 21 09:51:57 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 971, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: closed. This flag only has an
> effect if this :class:`.Session` is
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 403, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2112, in contextual_connect
> May 21 09:51:57 airmaster01 airflow[26451]:
> self._wrap_pool_connect(self.pool.connect, None),
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2151, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:57 airmaster01 airflow[26451]: exc_info
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 203, in raise_from_cause
> May 21 09:51:57 airmaster01 airflow[26451]: reraise(type(exception),
> exception, tb=exc_tb, cause=cause)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 186, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError:
> (psycopg2.OperationalError) could not translate host name
> "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait,
> self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
> line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above
> exception, another exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could
> not translate host name "airstorage01.example.com" to address: Name or
> service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the
> direct cause of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
> line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
> line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
> line 67, in wsgi
> May 21 09:51:58 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 65, in load
> May 21 09:51:58 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 52, in load_wsgiapp
> May 21 09:51:58 airmaster01 airflow[26451]: return
> util.import_app(self.app_uri)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line
> 366, in import_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:58 airmaster01 airflow[26451]: File "<string>", line 1, in
> <module>
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 161, in cached_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 60, in create_app
> May 21 09:51:58 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 1975, in <module>
> May 21 09:51:58 airmaster01 airflow[26451]: class
> ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 2057, in ChartModelView
> May 21 09:51:58 airmaster01 airflow[26451]:
> .group_by(models.Connection.conn_id)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2855, in __iter__
> May 21 09:51:58 airmaster01 airflow[26451]: return
> self._execute_and_instances(context)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2876, in _execute_and_instances
> May 21 09:51:58 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2885, in _get_bind_args
> May 21 09:51:58 airmaster01 airflow[26451]: **kw
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2867, in _connection_from_session
> May 21 09:51:58 airmaster01 airflow[26451]: conn =
> self.session.connection(**kw)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 966, in connection
> May 21 09:51:58 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 971, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: closed. This flag only has an
> effect if this :class:`.Session` is
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 403, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2112, in contextual_connect
> May 21 09:51:58 airmaster01 airflow[26451]:
> self._wrap_pool_connect(self.pool.connect, None),
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2151, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:58 airmaster01 airflow[26451]: exc_info
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 203, in raise_from_cause
> May 21 09:51:58 airmaster01 airflow[26451]: reraise(type(exception),
> exception, tb=exc_tb, cause=cause)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 186, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: return fn()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:58 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:58 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:58 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:58 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:58 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:58 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError:
> (psycopg2.OperationalError) could not translate host name
> "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000]
> [23794] [INFO] Worker exiting (pid: 23794)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23561] [INFO] Worker exiting (pid: 23561)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23759] [INFO] Worker exiting (pid: 23759)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23620] [INFO] Worker exiting (pid: 23620)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23689] [INFO] Worker exiting (pid: 23689)
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000]
> [26464] [INFO] Shutting down: Master
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000]
> [26464] [INFO] Reason: Worker failed to boot.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
