Note: If I disable pickling dags, it all works: donot_pickle = True Though I don't understand what might be causing the pickling issue considering I am running everything on the same machine.
On Thu, Jun 16, 2016 at 5:22 PM, Vineet Goel <vin...@robinhood.com> wrote: > Hi, > > I am new to Airflow. I set up a local Airflow instance with > CeleryExecutor. When I execute the command: > > airflow backfill remote_execution_test -s 2014-07-01 -e 2014-07-02 > > > I get the following error in the worker logs: > > [2016-06-16 17:03:48,753: ERROR/Worker-9] Command 'airflow run > remote_execution_test print_date 2014-07-01T00:00:00 --pickle 7 --local ' > returned non-zero exit status 1 > > with the following stack trace: > > Traceback (most recent call last): > File "/Users/vineet/.virtualenvs/airflow/bin/airflow", line 15, in > <module> > args.func(args) > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/airflow/bin/cli.py", > line 211, in run > DagPickle).filter(DagPickle.id == args.pickle).first() > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/query.py", > line 2659, in first > ret = list(self[0:1]) > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/query.py", > line 2457, in __getitem__ > return list(res) > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", > line 86, in instances > util.raise_from_cause(err) > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/util/compat.py", > line 202, in raise_from_cause > reraise(type(exception), exception, tb=exc_tb, cause=cause) > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", > line 71, in instances > rows = [proc(row) for row in fetch] > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", > line 428, in _instance > loaded_instance, populate_existing, populators) > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", > line 486, in _populate_full > dict_[key] = getter(row) > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/sqlalchemy/sql/sqltypes.py", > line 1253, in process > return loads(value) > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/dill/dill.py", > line 260, in loads > return load(file) > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/dill/dill.py", > line 250, in load > obj = pik.load() > File > "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", > line 858, in load > dispatch[key](self) > File > "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", > line 1090, in load_global > klass = self.find_class(module, name) > File > "/Users/vineet/.virtualenvs/airflow/lib/python2.7/site-packages/dill/dill.py", > line 406, in find_class > return StockUnpickler.find_class(self, module, name) > File > "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", > line 1124, in find_class > __import__(module) > ImportError: No module named unusual_prefix_remote_execution_test > > I am trying to understand what I might be doing wrong here. I tried to run > the command that celery is trying to run without the --pickle flag and it > works. > > Note: I am running a single worker on the same machine. > > Thanks! >
