You might need to pass the --save_main_session pipeline option. See: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#pickling-and-managing-the-main-session
Alternatively, structure your pipeline as a package: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#nonpython On Fri, Feb 23, 2024 at 6:14 PM Juan Romero <[email protected]> wrote: > Hi guys. > > Currently i have the class SaptmCustomerUpsertPipeline that inherited from > the GenericPipeline class. Locally the pipeline works fine, but when i try > to run it in dataflow i got this error: > > severity: "ERROR" > textPayload: "Error message from worker: generic::unknown: Traceback > (most recent call last): File > "/usr/local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py", > line 285, in loads return dill.loads(s) File > "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 275, in loads > return load(file, ignore, **kwds) File > "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 270, in load > return Unpickler(file, ignore=ignore, **kwds).load() File > "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 472, in load > obj = StockUnpickler.load(self) File > "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 462, in > find_class return StockUnpickler.find_class(self, module, name) > AttributeError: Can't get attribute 'SaptmCustomerUpsertPipeline' on > <module 'apache_beam.runners.worker.sdk_worker_main' from > '/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker_main.py'> > > Seems that there is an error in the serialization of the child. class. > Someone has faced with this problem? . The pipeline works fine with the > direct runner. > >
