Hi Pablo, This is strange... it doesn't seem to be the last beam release as last night it was already using 2.19.0 I wonder if it was some release from the DataFlow team (not beam related): Job typeBatch Job status Succeeded SDK version Apache Beam Python 3.5 SDK 2.19.0 Region us-central1 Start timeFebruary 3, 2020 at 9:28:35 PM GMT-8 Elapsed time5 min 11 sec
On Tue, Feb 4, 2020 at 9:15 AM Pablo Estrada <pabl...@google.com> wrote: > Hi Alan, > could it be that you're picking up the new Apache Beam 2.19.0 release? > Could you try depending on beam 2.18.0 to see if the issue surfaces when > using the new release? > > If something was working and no longer works, it sounds like a bug. This > may have to do with how we pickle (dill / cloudpickle) - see this question > https://stackoverflow.com/questions/42960637/python-3-5-dill-pickling-unpickling-on-different-servers-keyerror-classtype > Best > -P. > > On Tue, Feb 4, 2020 at 6:22 AM Alan Krumholz <alan.krumh...@betterup.co> > wrote: > >> Hi, >> >> I was running a dataflow job in GCP last night and it was running fine. >> This morning this same exact job is failing with the following error: >> >> Error message from worker: Traceback (most recent call last): File >> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", >> line 286, in loads return dill.loads(s) File >> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 275, in loads >> return load(file, ignore, **kwds) File >> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 270, in load >> return Unpickler(file, ignore=ignore, **kwds).load() File >> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 472, in load >> obj = StockUnpickler.load(self) File >> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 577, in >> _load_type return _reverse_typemap[name] KeyError: 'ClassType' During >> handling of the above exception, another exception occurred: Traceback >> (most recent call last): File >> "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", >> line 648, in do_work work_executor.execute() File >> "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line >> 176, in execute op.start() File "apache_beam/runners/worker/operations.py", >> line 649, in apache_beam.runners.worker.operations.DoOperation.start File >> "apache_beam/runners/worker/operations.py", line 651, in >> apache_beam.runners.worker.operations.DoOperation.start File >> "apache_beam/runners/worker/operations.py", line 652, in >> apache_beam.runners.worker.operations.DoOperation.start File >> "apache_beam/runners/worker/operations.py", line 261, in >> apache_beam.runners.worker.operations.Operation.start File >> "apache_beam/runners/worker/operations.py", line 266, in >> apache_beam.runners.worker.operations.Operation.start File >> "apache_beam/runners/worker/operations.py", line 597, in >> apache_beam.runners.worker.operations.DoOperation.setup File >> "apache_beam/runners/worker/operations.py", line 602, in >> apache_beam.runners.worker.operations.DoOperation.setup File >> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", >> line 290, in loads return dill.loads(s) File >> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 275, in loads >> return load(file, ignore, **kwds) File >> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 270, in load >> return Unpickler(file, ignore=ignore, **kwds).load() File >> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 472, in load >> obj = StockUnpickler.load(self) File >> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 577, in >> _load_type return _reverse_typemap[name] KeyError: 'ClassType' >> >> >> If I use a local runner it still runs fine. >> Anyone else experiencing something similar today? (or know how to fix >> this?) >> >> Thanks! >> >
