davidcavazos commented on issue #24458: URL: https://github.com/apache/beam/issues/24458#issuecomment-1374943005
I also have a similar issue. I'm trying to install HuggingFace [`datasets`](https://huggingface.co/docs/datasets/index) which depends on `multiprocess`, so switching to `multiprocessing` is not an option. Right now `dill` is pinned to `0.3.1.1` in Beam, which is from 2019 and `datasets` 2.x is from 2022 and is pinning to the latest version of `multiprocess` available, so it's impossible to make it work. Could it be an option to pin each Beam release with the latest version available of `dill` similar to what `multiprocess` does? That would also benefit of any improvements and bug fixes they make. Since both server and workers would have the same Beam version, they should also have the same `dill` version, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
