Hi Alan, +Valentyn Tymofieiev <valen...@google.com> Can you verify if my assumption is correct?
It seems that the problem might come from dill version mismatch. Dill version should match on worker and user code. Between Beam 2.17 and Beam 2.18 we upgraded dill version to 0.3.1.1 which has an incompatible format with earlier versions. Which version of dill do you use when submitting pipeline? Try using dill version below 0.3.1 with Beam 2.17 and earlier. And Dill 0.3.1 or above with Beam 2.18 and above. Regards, --Mikhail. On Tue, Feb 4, 2020 at 10:04 AM Alan Krumholz <alan.krumh...@betterup.co> wrote: > I tried breaking apart my pipeline. Seems the step that breaks it is: > beam.io.WriteToBigQuery > > Let me see if I can create a self contained example that breaks to share > with you > > Thanks! > > On Tue, Feb 4, 2020 at 9:53 AM Pablo Estrada <pabl...@google.com> wrote: > >> Hm that's odd. No changes to the pipeline? Are you able to share some of >> the code? >> >> +Udi Meiri <eh...@google.com> do you have any idea what could be going >> on here? >> >> On Tue, Feb 4, 2020 at 9:25 AM Alan Krumholz <alan.krumh...@betterup.co> >> wrote: >> >>> Hi Pablo, >>> This is strange... it doesn't seem to be the last beam release as last >>> night it was already using 2.19.0 I wonder if it was some release from the >>> DataFlow team (not beam related): >>> Job typeBatch >>> Job status Succeeded >>> SDK version >>> Apache Beam Python 3.5 SDK 2.19.0 >>> Region >>> us-central1 >>> Start timeFebruary 3, 2020 at 9:28:35 PM GMT-8 >>> Elapsed time5 min 11 sec >>> >>> On Tue, Feb 4, 2020 at 9:15 AM Pablo Estrada <pabl...@google.com> wrote: >>> >>>> Hi Alan, >>>> could it be that you're picking up the new Apache Beam 2.19.0 release? >>>> Could you try depending on beam 2.18.0 to see if the issue surfaces when >>>> using the new release? >>>> >>>> If something was working and no longer works, it sounds like a bug. >>>> This may have to do with how we pickle (dill / cloudpickle) - see this >>>> question >>>> https://stackoverflow.com/questions/42960637/python-3-5-dill-pickling-unpickling-on-different-servers-keyerror-classtype >>>> Best >>>> -P. >>>> >>>> On Tue, Feb 4, 2020 at 6:22 AM Alan Krumholz <alan.krumh...@betterup.co> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I was running a dataflow job in GCP last night and it was running fine. >>>>> This morning this same exact job is failing with the following error: >>>>> >>>>> Error message from worker: Traceback (most recent call last): File >>>>> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", >>>>> line 286, in loads return dill.loads(s) File >>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 275, in loads >>>>> return load(file, ignore, **kwds) File >>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 270, in load >>>>> return Unpickler(file, ignore=ignore, **kwds).load() File >>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 472, in load >>>>> obj = StockUnpickler.load(self) File >>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 577, in >>>>> _load_type return _reverse_typemap[name] KeyError: 'ClassType' During >>>>> handling of the above exception, another exception occurred: Traceback >>>>> (most recent call last): File >>>>> "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", >>>>> line 648, in do_work work_executor.execute() File >>>>> "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line >>>>> 176, in execute op.start() File >>>>> "apache_beam/runners/worker/operations.py", >>>>> line 649, in apache_beam.runners.worker.operations.DoOperation.start File >>>>> "apache_beam/runners/worker/operations.py", line 651, in >>>>> apache_beam.runners.worker.operations.DoOperation.start File >>>>> "apache_beam/runners/worker/operations.py", line 652, in >>>>> apache_beam.runners.worker.operations.DoOperation.start File >>>>> "apache_beam/runners/worker/operations.py", line 261, in >>>>> apache_beam.runners.worker.operations.Operation.start File >>>>> "apache_beam/runners/worker/operations.py", line 266, in >>>>> apache_beam.runners.worker.operations.Operation.start File >>>>> "apache_beam/runners/worker/operations.py", line 597, in >>>>> apache_beam.runners.worker.operations.DoOperation.setup File >>>>> "apache_beam/runners/worker/operations.py", line 602, in >>>>> apache_beam.runners.worker.operations.DoOperation.setup File >>>>> "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", >>>>> line 290, in loads return dill.loads(s) File >>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 275, in loads >>>>> return load(file, ignore, **kwds) File >>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 270, in load >>>>> return Unpickler(file, ignore=ignore, **kwds).load() File >>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 472, in load >>>>> obj = StockUnpickler.load(self) File >>>>> "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 577, in >>>>> _load_type return _reverse_typemap[name] KeyError: 'ClassType' >>>>> >>>>> >>>>> If I use a local runner it still runs fine. >>>>> Anyone else experiencing something similar today? (or know how to fix >>>>> this?) >>>>> >>>>> Thanks! >>>>> >>>>
