tvalentyn opened a new issue, #23119:
URL: https://github.com/apache/beam/issues/23119
### What needs to happen?
If a Beam dependency has a flexible upper bound, users will download the
most recent compatible version of a dependency at sdk installation time.
Overtime, the version used at job submission may become newer than the version
installed in a released Beam container. Given that forwards-compatiblity of a
pickle library is not guaranteed, the pipeline may fail to unpickle at runtime.
To mitigate, pickle libraries should be specified in `install_requires` with
tight version bounds limiting to a particular minor version. This can cause
inconvenience to Beam user, if we depend on an old version of a library.
Therefore, we should periodically update the version we use, at least once per
release cycle.
We can consider to close this issue when either condition is met:
- Beam vendors cloudpickle.
- Beam stages used version of cloudpickle at pipeline submission time, and
uses it at runtime.
- Beam communicates to the users a list of vetted versions of each Beam
dependency that users must install to use Beam in a supported configuration.
Then, we can open up a version range.
Until then, don't close this issue, instead, move it to the next release
milestone.
### Issue Priority
Priority: 3
### Issue Component
Component: sdk-py-core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
