Swapnil Debarshi created AIRFLOW-2676:
-----------------------------------------
Summary: Import Error with Celery Executor
Key: AIRFLOW-2676
URL: https://issues.apache.org/jira/browse/AIRFLOW-2676
Project: Apache Airflow
Issue Type: Bug
Components: worker
Affects Versions: 1.9.0
Environment: MacOS High Sierra, Linux Ubuntu 16.04
Reporter: Swapnil Debarshi
We wanted to build our own Airflow distribution and built v1.9.0 from git as
well as the zip distribution with `python setup.py install`. While using the
build in a local cluster, everything works fine. But when setting up in a
docker cluster with docker compose or as a marathon app on a Mesos cluster
using the CeleryExecutor, I get an error in the following error in the worker
logs while using the PythonOperator:
{code:python}
ModuleNotFoundError: No module named
'unusual_prefix_3446424b361fa6ae6ad938f278ea04d10cf4cd14_test_failure'
{code}
Testing the BashOperator works just fine. Can also confirm it does not throw
any error when I install Airflow using `pip` or running the celery worker
locally along with the other airflow components. The `docker-compose.yml` can
be compared to
[here|https://github.com/puckel/docker-airflow/blob/master/docker-compose-CeleryExecutor.yml]
The scheduler log is as follows:
{code:python}
[2018-06-21 20:32:26,031] {jobs.py:1292} INFO - Sending ('test_failure',
'date', datetime.datetime(2018, 6, 20, 20, 31, 53, 663786)) to executor with
priority 1 and queue default
[2018-06-21 20:32:26,031] {base_executor.py:49} INFO - Adding to queue: airflow
run test_failure date 2018-06-20T20:31:53.663786 --pickle 10 --local -sd
/usr/local/airflow/dags/test_failure.py
{code}
The worker log is as follows:
{code:python}
[2018-06-21 20:32:34,175] {celery_executor.py:50} INFO - Executing command in
Celery: airflow run test_failure date 2018-06-20T20:31:53.663786 --pickle 10
--local -sd /usr/local/airflow/dags/test_failure.py
[2018-06-21 20:32:35,861] {configuration.py:206} WARNING - section/key
[celery/celery_ssl_active] not found in config
[2018-06-21 20:32:35,861] {default_celery.py:41} WARNING - Celery Executor will
run without SSL
[2018-06-21 20:32:35,862] {__init__.py:45} INFO - Using executor CeleryExecutor
[2018-06-21 20:32:35,978] {cli.py:356} INFO - Loading pickle id 10
[2018-06-21 20:32:36,135] {celery_executor.py:54} ERROR - Command 'airflow run
test_failure date 2018-06-20T20:31:53.663786 --pickle 10 --local -sd
/usr/local/airflow/dags/test_failure.py' returned non-zero exit status 1.
{code}
The worker stack trace is as follows:
{code:python}
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 4, in <module>
__import__('pkg_resources').run_script('apache-airflow==1.9.0', 'airflow')
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line
654, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line
1434, in run_script
exec(code, namespace, namespace)
File
"/usr/local/lib/python3.6/site-packages/apache_airflow-1.9.0-py3.6.egg/EGG-INFO/scripts/airflow",
line 27, in <module>
args.func(args)
File
"/usr/local/lib/python3.6/site-packages/apache_airflow-1.9.0-py3.6.egg/airflow/bin/cli.py",
line 358, in run
DagPickle).filter(DagPickle.id == args.pickle).first()
File
"/usr/local/lib/python3.6/site-packages/SQLAlchemy-1.2.8-py3.6-linux-x86_64.egg/sqlalchemy/orm/query.py",
line 2835, in first
ret = list(self[0:1])
File
"/usr/local/lib/python3.6/site-packages/SQLAlchemy-1.2.8-py3.6-linux-x86_64.egg/sqlalchemy/orm/query.py",
line 2627, in __getitem__
return list(res)
File
"/usr/local/lib/python3.6/site-packages/SQLAlchemy-1.2.8-py3.6-linux-x86_64.egg/sqlalchemy/orm/loading.py",
line 98, in instances
util.raise_from_cause(err)
File
"/usr/local/lib/python3.6/site-packages/SQLAlchemy-1.2.8-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py",
line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File
"/usr/local/lib/python3.6/site-packages/SQLAlchemy-1.2.8-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py",
line 187, in reraise
raise value
File
"/usr/local/lib/python3.6/site-packages/SQLAlchemy-1.2.8-py3.6-linux-x86_64.egg/sqlalchemy/orm/loading.py",
line 79, in instances
rows = [proc(row) for row in fetch]
File
"/usr/local/lib/python3.6/site-packages/SQLAlchemy-1.2.8-py3.6-linux-x86_64.egg/sqlalchemy/orm/loading.py",
line 79, in <listcomp>
rows = [proc(row) for row in fetch]
File
"/usr/local/lib/python3.6/site-packages/SQLAlchemy-1.2.8-py3.6-linux-x86_64.egg/sqlalchemy/orm/loading.py",
line 511, in _instance
loaded_instance, populate_existing, populators)
File
"/usr/local/lib/python3.6/site-packages/SQLAlchemy-1.2.8-py3.6-linux-x86_64.egg/sqlalchemy/orm/loading.py",
line 611, in _populate_full
dict_[key] = getter(row)
File
"/usr/local/lib/python3.6/site-packages/SQLAlchemy-1.2.8-py3.6-linux-x86_64.egg/sqlalchemy/sql/sqltypes.py",
line 1588, in process
return loads(value)
File
"/usr/local/lib/python3.6/site-packages/dill-0.2.8.1-py3.6.egg/dill/_dill.py",
line 316, in loads
return load(file, ignore)
File
"/usr/local/lib/python3.6/site-packages/dill-0.2.8.1-py3.6.egg/dill/_dill.py",
line 304, in load
obj = pik.load()
File
"/usr/local/lib/python3.6/site-packages/dill-0.2.8.1-py3.6.egg/dill/_dill.py",
line 464, in find_class
return StockUnpickler.find_class(self, module, name)
ModuleNotFoundError: No module named
'unusual_prefix_3446424b361fa6ae6ad938f278ea04d10cf4cd14_test_failure'
[2018-06-21 20:32:36,141: ERROR/ForkPoolWorker-1] Task
airflow.executors.celery_executor.execute_command[0d89989b-850a-40be-ae7c-7886fe881b4b]
raised unexpected: AirflowException('Celery command failed',)
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/site-packages/apache_airflow-1.9.0-py3.6.egg/airflow/executors/celery_executor.py",
line 52, in execute_command
subprocess.check_call(command, shell=True)
File "/usr/local/lib/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'airflow run test_failure date
2018-06-20T20:31:53.663786 --pickle 10 --local -sd
/usr/local/airflow/dags/test_failure.py' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 375, in
trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 632, in
__protected_call__
return self.run(*args, **kwargs)
File
"/usr/local/lib/python3.6/site-packages/apache_airflow-1.9.0-py3.6.egg/airflow/executors/celery_executor.py",
line 55, in execute_command
raise AirflowException('Celery command failed')
airflow.exceptions.AirflowException: Celery command failed
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning:
The psycopg2 wheel package will be renamed from release 2.8; in order to keep
installing from binary please use "pip install psycopg2-binary" instead. For
details see:
<http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning:
The psycopg2 wheel package will be renamed from release 2.8; in order to keep
installing from binary please use "pip install psycopg2-binary" instead. For
details see:
<http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning:
The psycopg2 wheel package will be renamed from release 2.8; in order to keep
installing from binary please use "pip install psycopg2-binary" instead. For
details see:
<http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning:
The psycopg2 wheel package will be renamed from release 2.8; in order to keep
installing from binary please use "pip install psycopg2-binary" instead. For
details see:
<http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
