riteshghorse opened a new issue, #27653:
URL: https://github.com/apache/beam/issues/27653
### What happened?
Running python pipelines with `TFModelHandlerTensor` when used with
`load_model_args` got some pickling issues on Direct Runner. Haven't verified
on other runners
```
File
"/home/student_00_40bc0e514d9c/devrel-demos/data-analytics/beam_ml_toxicity_in_gaming/part1.py",
line 141, in <module>
run(known_args.project_id, known_args.gaming_loc, known_args.movie_loc,
pipeline_args)
File
"/home/student_00_40bc0e514d9c/devrel-demos/data-analytics/beam_ml_toxicity_in_gaming/part1.py",
line 117, in run
_ = (not_filter
File
"/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/pipeline.py",
line 600, in __exit__
self.result = self.run()
File
"/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/pipeline.py",
line 574, in run
pickler.dump_session(os.path.join(tmpdir, 'main_session.pickle'))
File
"/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/internal/pickler.py",
line 61, in dump_session
return desired_pickle_lib.dump_session(file_path)
File
"/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py",
line 439, in dump_session
dill.dump_session(file_path)
File
"/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/dill/_dill.py",
line 351, in dump_session
pickler.dump(main)
File
"/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/dill/_dill.py",
line 445, in dump
StockPickler.dump(self, obj)
File "/usr/lib/python3.9/pickle.py", line 487, in dump
self.save(obj)
File "/usr/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File
"/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py",
line 301, in save_module
return old_save_module(pickler, obj)
File
"/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/dill/_dill.py",
line 1294, in save_module
pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
File "/usr/lib/python3.9/pickle.py", line 717, in save_reduce
save(state)
File "/usr/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File
"/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py",
line 349, in new_save_module_dict
return old_save_module_dict(pickler, obj)
File
"/home/student_00_40bc0e514d9c/.local/lib/python3.9/site-packages/dill/_dill.py",
line 912, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.9/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.9/pickle.py", line 997, in _batch_setitems
save(v)
File "/usr/lib/python3.9/pickle.py", line 578, in save
rv = reduce(self.proto)
TypeError: cannot pickle 'LazyLoader' object
```
### Issue Priority
Priority: 3 (minor)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
