yeandy opened a new pull request, #22001: URL: https://github.com/apache/beam/pull/22001
`dill` was bumped to 0.3.5.1 last month. With this version, there were no issues pickling the classes defined in `pytorch_language_modeling.py` during development. However, since we only recently reverted `dill` back to `0.3.1.1`, and haven't had a chance to test `pytorch_language_modeling.py` till `apache_beam==2.40.0rc1` was released, we didn't realize that dill version `0.3.1.1` has issues with pickling some of the classes. See error [here](https://gist.github.com/yeandy/4c51243213e24eef9417ff386b24cd0e) **Solution**: Move the wrapper class outside of the `run()` function. This example now works for the current dill package `dill==0.3.1.1`. Note: I tested a hacky solution by adding the following code to `pytorch_language_modeling.py`. ``` from apache_beam.internal import pickler pickler.set_library('cloudpickle') ``` But this is not good practice, so I went with the change in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
