[
https://issues.apache.org/jira/browse/AIRFLOW-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022892#comment-16022892
]
Kengo Seki commented on AIRFLOW-1235:
-------------------------------------
I could reproduce this issue as follows:
1. Run webserver in foreground
2. Kill gunicorn master
{code}
$ ps aux | grep airflow | grep -v grep
sekikn 2929 4.5 9.3 241204 71172 pts/1 S+ 09:12 0:02
/home/sekikn/.virtualenvs/a/bin/python2 /home/sekikn/.virtualenvs/a/bin/airflow
webserver
sekikn 2935 1.9 6.1 115868 46676 pts/1 S+ 09:12 0:00 gunicorn:
master [airflow-webserver]
sekikn 2943 1.3 8.8 242144 67300 pts/1 Sl+ 09:12 0:00 [ready]
gunicorn: worker [airflow-webserver]
sekikn 2944 1.3 8.8 242040 67312 pts/1 Sl+ 09:12 0:00 [ready]
gunicorn: worker [airflow-webserver]
sekikn 2945 1.3 8.8 242052 67320 pts/1 Sl+ 09:12 0:00 [ready]
gunicorn: worker [airflow-webserver]
sekikn 2952 6.0 8.8 242056 67196 pts/1 Sl+ 09:13 0:00 [ready]
gunicorn: worker [airflow-webserver]
$ kill 2935
{code}
3. Then gunicorn master remains as a zombie and webserver gets stuck
{code}
$ ps aux | grep airflow | grep -v grep
sekikn 2929 10.5 9.3 241204 71220 pts/1 S+ 09:12 0:22
/home/sekikn/.virtualenvs/a/bin/python2 /home/sekikn/.virtualenvs/a/bin/airflow
webserver
$ ps 2935
PID TTY STAT TIME COMMAND
2935 pts/1 Z+ 0:00 [gunicorn: maste] <defunct>
{code}
At step 3, the following message is output to the log:
{code}
[2017-05-24 09:13:52,092] [2929] {cli.py:671} DEBUG - [4 / 4] doing a refresh
of 1 workers
{code}
So I think webserver waits for workers infinitely (but they will never start)
at line 679.
{code}
668 def start_refresh(gunicorn_master_proc):
669 batch_size = conf.getint('webserver', 'worker_refresh_batch_size')
670 logging.debug('%s doing a refresh of %s workers',
671 state, batch_size)
672 sys.stdout.flush()
673 sys.stderr.flush()
674
675 excess = 0
676 for _ in range(batch_size):
677 gunicorn_master_proc.send_signal(signal.SIGTTIN)
678 excess += 1
679 wait_until_true(lambda: num_workers_expected + excess ==
680 get_num_workers_running(gunicorn_master_proc))
{code}
> Odd behaviour when all gunicorn workers die
> -------------------------------------------
>
> Key: AIRFLOW-1235
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1235
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
> Affects Versions: 1.8.0
> Reporter: Erik Forsberg
>
> The webserver has sometimes stopped responding to port 443, and today I found
> the issue - I had a misconfigured resolv.conf that made it unable to talk to
> my postgresql. This was the root cause, but the way airflow webserver behaved
> was a bit odd.
> It seems that when all gunicorn workers failed to start, the gunicorn master
> shut down. However, the main process (the one that starts gunicorn master)
> did not shut down, so there was no way of detecting the failed status of
> webserver from e.g. systemd or init script.
> Full traceback leading to stale webserver process:
> {noformat}
> May 21 09:51:57 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000]
> [23794] [ERROR] Exception in worker process:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait,
> self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
> line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above
> exception, another exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could
> not translate host name "airstorage01.example.com" to address: Name or
> service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the
> direct cause of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
> line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
> line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
> line 67, in wsgi
> May 21 09:51:57 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 65, in load
> May 21 09:51:57 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 52, in load_wsgiapp
> May 21 09:51:57 airmaster01 airflow[26451]: return
> util.import_app(self.app_uri)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line
> 366, in import_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:57 airmaster01 airflow[26451]: File "<string>", line 1, in
> <module>
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 161, in cached_app
> May 21 09:51:57 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 60, in create_app
> May 21 09:51:57 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 1975, in <module>
> May 21 09:51:57 airmaster01 airflow[26451]: class
> ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 2057, in ChartModelView
> May 21 09:51:57 airmaster01 airflow[26451]:
> .group_by(models.Connection.conn_id)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2855, in __iter__
> May 21 09:51:57 airmaster01 airflow[26451]: return
> self._execute_and_instances(context)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2876, in _execute_and_instances
> May 21 09:51:57 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2885, in _get_bind_args
> May 21 09:51:57 airmaster01 airflow[26451]: **kw
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2867, in _connection_from_session
> May 21 09:51:57 airmaster01 airflow[26451]: conn =
> self.session.connection(**kw)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 966, in connection
> May 21 09:51:57 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 971, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: closed. This flag only has an
> effect if this :class:`.Session` is
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 403, in _connection_for_bind
> May 21 09:51:57 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2112, in contextual_connect
> May 21 09:51:57 airmaster01 airflow[26451]:
> self._wrap_pool_connect(self.pool.connect, None),
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2151, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:57 airmaster01 airflow[26451]: exc_info
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 203, in raise_from_cause
> May 21 09:51:57 airmaster01 airflow[26451]: reraise(type(exception),
> exception, tb=exc_tb, cause=cause)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 186, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError:
> (psycopg2.OperationalError) could not translate host name
> "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1122, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait,
> self._timeout)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py",
> line 145, in get
> May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
> May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
> May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above
> exception, another exception occurred:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:57 airmaster01 airflow[26451]: return fn()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:57 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:57 airmaster01 airflow[26451]: raise value
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:57 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:57 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could
> not translate host name "airstorage01.example.com" to address: Name or
> service not known
> May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the
> direct cause of the following exception:
> May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py",
> line 507, in spawn_worker
> May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py",
> line 118, in init_process
> May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
> May 21 09:51:57 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py",
> line 67, in wsgi
> May 21 09:51:58 airmaster01 airflow[26451]: self.callable = self.load()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 65, in load
> May 21 09:51:58 airmaster01 airflow[26451]: return self.load_wsgiapp()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py",
> line 52, in load_wsgiapp
> May 21 09:51:58 airmaster01 airflow[26451]: return
> util.import_app(self.app_uri)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line
> 366, in import_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
> May 21 09:51:58 airmaster01 airflow[26451]: File "<string>", line 1, in
> <module>
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 161, in cached_app
> May 21 09:51:58 airmaster01 airflow[26451]: app = create_app(config)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py",
> line 60, in create_app
> May 21 09:51:58 airmaster01 airflow[26451]: from airflow.www import views
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 1975, in <module>
> May 21 09:51:58 airmaster01 airflow[26451]: class
> ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py",
> line 2057, in ChartModelView
> May 21 09:51:58 airmaster01 airflow[26451]:
> .group_by(models.Connection.conn_id)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2855, in __iter__
> May 21 09:51:58 airmaster01 airflow[26451]: return
> self._execute_and_instances(context)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2876, in _execute_and_instances
> May 21 09:51:58 airmaster01 airflow[26451]: close_with_result=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2885, in _get_bind_args
> May 21 09:51:58 airmaster01 airflow[26451]: **kw
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
> line 2867, in _connection_from_session
> May 21 09:51:58 airmaster01 airflow[26451]: conn =
> self.session.connection(**kw)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 966, in connection
> May 21 09:51:58 airmaster01 airflow[26451]: cannot otherwise be identified.
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 971, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: closed. This flag only has an
> effect if this :class:`.Session` is
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
> line 403, in _connection_for_bind
> May 21 09:51:58 airmaster01 airflow[26451]: conn = bind.contextual_connect()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2112, in contextual_connect
> May 21 09:51:58 airmaster01 airflow[26451]:
> self._wrap_pool_connect(self.pool.connect, None),
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2151, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: e, dialect, self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 1465, in _handle_dbapi_exception_noconnection
> May 21 09:51:58 airmaster01 airflow[26451]: exc_info
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 203, in raise_from_cause
> May 21 09:51:58 airmaster01 airflow[26451]: reraise(type(exception),
> exception, tb=exc_tb, cause=cause)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 186, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value.with_traceback(tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
> line 2147, in _wrap_pool_connect
> May 21 09:51:58 airmaster01 airflow[26451]: return fn()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 387, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return
> _ConnectionFairy._checkout(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 766, in _checkout
> May 21 09:51:58 airmaster01 airflow[26451]: fairy =
> _ConnectionRecord.checkout(pool)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 516, in checkout
> May 21 09:51:58 airmaster01 airflow[26451]: rec = pool._do_get()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1138, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: self._dec_overflow()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> May 21 09:51:58 airmaster01 airflow[26451]: compat.reraise(exc_type,
> exc_value, exc_tb)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> May 21 09:51:58 airmaster01 airflow[26451]: raise value
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 1135, in _do_get
> May 21 09:51:58 airmaster01 airflow[26451]: return self._create_connection()
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 333, in _create_connection
> May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionRecord(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 461, in __init__
> May 21 09:51:58 airmaster01 airflow[26451]:
> self.__connect(first_connect_check=True)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py",
> line 651, in __connect
> May 21 09:51:58 airmaster01 airflow[26451]: connection =
> pool._invoke_creator(self)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py",
> line 105, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return dialect.connect(*cargs,
> **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
> line 393, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs,
> **cparams)
> May 21 09:51:58 airmaster01 airflow[26451]: File
> "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py",
> line 130, in connect
> May 21 09:51:58 airmaster01 airflow[26451]: conn = _connect(dsn,
> connection_factory=connection_factory, **kwasync)
> May 21 09:51:58 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError:
> (psycopg2.OperationalError) could not translate host name
> "airstorage01.example.com" to address: Name or service not known
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000]
> [23794] [INFO] Worker exiting (pid: 23794)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23561] [INFO] Worker exiting (pid: 23561)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23759] [INFO] Worker exiting (pid: 23759)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23620] [INFO] Worker exiting (pid: 23620)
> May 21 09:51:58 airmaster01 airflow[26451]:
> /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71:
> ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use
> flask_cache instead.
> May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname),
> ExtDeprecationWarning
> May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000]
> [23689] [INFO] Worker exiting (pid: 23689)
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000]
> [26464] [INFO] Shutting down: Master
> May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000]
> [26464] [INFO] Reason: Worker failed to boot.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
