Hm that's odd. No changes to the pipeline? Are you able to share some of the code?
+Udi Meiri <eh...@google.com> do you have any idea what could be going on here? On Tue, Feb 4, 2020 at 9:25 AM Alan Krumholz <alan.krumh...@betterup.co> wrote: > Hi Pablo, > This is strange... it doesn't seem to be the last beam release as last > night it was already using 2.19.0 I wonder if it was some release from the > DataFlow team (not beam related): > Job typeBatch > Job status Succeeded > SDK version > Apache Beam Python 3.5 SDK 2.19.0 > Region > us-central1 > Start timeFebruary 3, 2020 at 9:28:35 PM GMT-8 > Elapsed time5 min 11 sec > > On Tue, Feb 4, 2020 at 9:15 AM Pablo Estrada <pabl...@google.com> wrote: > >> Hi Alan, >> could it be that you're picking up the new Apache Beam 2.19.0 release? >> Could you try depending on beam 2.18.0 to see if the issue surfaces when >> using the new release? >> >> If something was working and no longer works, it sounds like a bug. This >> may have to do with how we pickle (dill / cloudpickle) - see this question >> https://stackoverflow.com/questions/42960637/python-3-5-dill-pickling-unpickling-on-different-servers-keyerror-classtype >> Best >> -P. >> >> On Tue, Feb 4, 2020 at 6:22 AM Alan Krumholz <alan.krumh...@betterup.co> >> wrote: >> >>> Hi, >>> >>> I was running a dataflow job in GCP last night and it was running fine. >>> This morning this same exact job is failing with the following error: >>> >>> Error message from worker: Traceback (most recent call last): File >>> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", >>> line 286, in loads return dill.loads(s) File >>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 275, in loads >>> return load(file, ignore, **kwds) File >>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 270, in load >>> return Unpickler(file, ignore=ignore, **kwds).load() File >>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 472, in load >>> obj = StockUnpickler.load(self) File >>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 577, in >>> _load_type return _reverse_typemap[name] KeyError: 'ClassType' During >>> handling of the above exception, another exception occurred: Traceback >>> (most recent call last): File >>> "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", >>> line 648, in do_work work_executor.execute() File >>> "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line >>> 176, in execute op.start() File "apache_beam/runners/worker/operations.py", >>> line 649, in apache_beam.runners.worker.operations.DoOperation.start File >>> "apache_beam/runners/worker/operations.py", line 651, in >>> apache_beam.runners.worker.operations.DoOperation.start File >>> "apache_beam/runners/worker/operations.py", line 652, in >>> apache_beam.runners.worker.operations.DoOperation.start File >>> "apache_beam/runners/worker/operations.py", line 261, in >>> apache_beam.runners.worker.operations.Operation.start File >>> "apache_beam/runners/worker/operations.py", line 266, in >>> apache_beam.runners.worker.operations.Operation.start File >>> "apache_beam/runners/worker/operations.py", line 597, in >>> apache_beam.runners.worker.operations.DoOperation.setup File >>> "apache_beam/runners/worker/operations.py", line 602, in >>> apache_beam.runners.worker.operations.DoOperation.setup File >>> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", >>> line 290, in loads return dill.loads(s) File >>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 275, in loads >>> return load(file, ignore, **kwds) File >>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 270, in load >>> return Unpickler(file, ignore=ignore, **kwds).load() File >>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 472, in load >>> obj = StockUnpickler.load(self) File >>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 577, in >>> _load_type return _reverse_typemap[name] KeyError: 'ClassType' >>> >>> >>> If I use a local runner it still runs fine. >>> Anyone else experiencing something similar today? (or know how to fix >>> this?) >>> >>> Thanks! >>> >>
