[
https://issues.apache.org/jira/browse/AIRFLOW-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522805#comment-16522805
]
kunal gupta commented on AIRFLOW-2631:
--------------------------------------
[~ashb]
I have updated my Celery to 4.2:
{color:#205081}sh-4.2$ pip show celery {color}
{color:#205081}Name: celery {color}
{color:#205081}Version: 4.2.0 {color}
{color:#205081}Summary: Distributed Task Queue. {color}
{color:#205081}Home-page: http://celeryproject.org {color}
{color:#205081}Author: Ask Solem {color}
{color:#205081}Author-email: [email protected] {color}
{color:#205081}License: BSD {color}
{color:#205081}Location: /usr/lib/python2.7/site-packages {color}
{color:#205081}Requires: pytz, billiard, kombu {color}
{color:#205081}Required-by: flower{color}
{color:#333333}After doing this im getting this error from my worker:{color}
{color:#205081}[2018-06-25 15:56:32,413] \{__init__.py:51} INFO - Using
executor CeleryExecutor
[2018-06-25 15:56:32,428] \{plugins_manager.py:102} DEBUG - Creating module
airflow.operators.rest_api
[2018-06-25 15:56:32,428] \{plugins_manager.py:102} DEBUG - Creating module
airflow.sensors.rest_api
[2018-06-25 15:56:32,428] \{plugins_manager.py:102} DEBUG - Creating module
airflow.hooks.rest_api
[2018-06-25 15:56:32,428] \{plugins_manager.py:102} DEBUG - Creating module
airflow.executors.rest_api
[2018-06-25 15:56:32,428] \{plugins_manager.py:102} DEBUG - Creating module
airflow.macros.rest_api
[2018-06-25 15:56:32,502] \{cli_action_loggers.py:40} DEBUG - Adding <function
default_action_log at 0x7fd48dcca320> to pre execution callback
[2018-06-25 15:56:32,610] \{cli_action_loggers.py:64} DEBUG - Calling
callbacks: [<function default_action_log at 0x7fd48dcca320>]
/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py:470: Warning:
Data truncated for column 'dttm' at row 1
cursor.execute(statement, parameters)
Starting flask
[2018-06-25 15:56:32,753] \{_internal.py:88} INFO - * Running on
http://0.0.0.0:8793/ (Press CTRL+C to quit)
[2018-06-25 15:56:45,296] \{settings.py:146} DEBUG - Setting up DB connection
pool (PID 95)
[2018-06-25 15:56:45,388] \{settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=3600
[2018-06-25 15:56:45,698] \{settings.py:146} DEBUG - Setting up DB connection
pool (PID 105)
[2018-06-25 15:56:45,789] \{settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=3600
[2018-06-25 15:56:46,592] \{settings.py:146} DEBUG - Setting up DB connection
pool (PID 91)
[2018-06-25 15:56:46,599] \{settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=3600
[2018-06-25 15:56:46,788] \{settings.py:146} DEBUG - Setting up DB connection
pool (PID 102)
[2018-06-25 15:56:46,793] \{settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=3600
[2018-06-25 15:56:48,698] \{__init__.py:42} DEBUG - Cannot import due to
doesn't look like a module path
[2018-06-25 15:56:48,888] \{__init__.py:42} DEBUG - Cannot import due to
doesn't look like a module path
[2018-06-25 15:56:49,292] \{__init__.py:42} DEBUG - Cannot import due to
doesn't look like a module path
[2018-06-25 15:56:54,695: ERROR/MainProcess] Process 'ForkPoolWorker-16' pid:89
exited with 'signal 9 (SIGKILL)'
[2018-06-25 15:56:54,709: ERROR/MainProcess] Process 'ForkPoolWorker-12' pid:85
exited with 'signal 9 (SIGKILL)'
[2018-06-25 15:56:54,988] \{settings.py:146} DEBUG - Setting up DB connection
pool (PID 104)
[2018-06-25 15:56:54,988] \{settings.py:146} DEBUG - Setting up DB connection
pool (PID 103)
[2018-06-25 15:56:54,988] \{settings.py:146} DEBUG - Setting up DB connection
pool (PID 92)
[2018-06-25 15:56:54,988] \{settings.py:146} DEBUG - Setting up DB connection
pool (PID 93)
[2018-06-25 15:56:54,988] \{settings.py:146} DEBUG - Setting up DB connection
pool (PID 96)
[2018-06-25 15:56:54,993] \{settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=3600
[2018-06-25 15:56:54,993] \{settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=3600
[2018-06-25 15:56:54,993] \{settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=3600
[2018-06-25 15:56:54,993] \{settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=3600
[2018-06-25 15:56:54,994] \{settings.py:174} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=3600
[2018-06-25 15:56:54,892: WARNING/MainProcess] Failed operation _store_result.
Retrying 2 more times.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/celery/backends/database/__init__.py",
line 53, in _inner
return fun(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/celery/backends/database/__init__.py",
line 105, in _store_result
session = self.ResultSession()
File "/usr/lib/python2.7/site-packages/celery/backends/database/__init__.py",
line 99, in ResultSession
**self.engine_options)
File "/usr/lib/python2.7/site-packages/celery/backends/database/session.py",
line 60, in session_factory
self.prepare_models(engine)
File "/usr/lib/python2.7/site-packages/celery/backends/database/session.py",
line 55, in prepare_models
ResultModelBase.metadata.create_all(engine)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3949,
in create_all
tables=tables)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
1928, in _run_visitor
with self._optional_conn_ctx_manager(connection) as conn:
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
1921, in _optional_conn_ctx_manager
with self.contextual_connect() as conn:
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
2112, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
2151, in _wrap_pool_connect
e, dialect, self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
1465, in _handle_dbapi_exception_noconnection
exc_info
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203,
in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
2147, in _wrap_pool_connect
return fn()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 387, in
connect
return _ConnectionFairy._checkout(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 768, in
_checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in
checkout
rec = pool._do_get()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1231, in
_do_get
return self._create_connection()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 333, in
_create_connection
return _ConnectionRecord(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 461, in
__init__
self.__connect(first_connect_check=True)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 651, in
__connect
connection = pool._invoke_creator(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py",
line 105, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line
393, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in
Connect
return Connection(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 193, in
__init__
super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (_mysql_exceptions.OperationalError) (2002, "Can't connect to
local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)")
[2018-06-25 15:56:55,093: WARNING/MainProcess] Failed operation _store_result.
Retrying 1 more times.
Traceback (most recent call last):{color}
> Worker fails to connect with MySQL DB
> -------------------------------------
>
> Key: AIRFLOW-2631
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2631
> Project: Apache Airflow
> Issue Type: Bug
> Affects Versions: Airflow 2.0, 1.10
> Reporter: kunal gupta
> Priority: Critical
>
> I have containerized Airflow webserver, scheduler, flower and worker. I'm
> using MySQL for the backend and Redis for queuing. Everytime I spin up the
> worker node it throws the error below and all the Dags are stuck in running
> status.
> Logs from the Worker Node:
> File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 193,
> in __init__
> super(Connection, self).__init__(*args, **kwargs2)
> OperationalError: (_mysql_exceptions.OperationalError) (2002, "Can't connect
> to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)")
> [2018-06-15 10:21:56,005: ERROR/MainProcess] Pool callback raised exception:
> OperationalError('(_mysql_exceptions.OperationalError) (2002, "Can\'t connect
> to local MySQL server through socket \'/var/lib/mysql/mysql.sock\' (2)")',)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
