claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418018720
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
But to answer the original question: I made this a flag so that if someone
knows they arent triggering the code path where the type is pickled using
coders and are doing a pipeline update they don't need dill 0.3.1.1 installed.
This can happen if they have a DeterministicFastPrimitivesCoder in the pipeline
but their key is not a "special type" e.g.
https://github.com/apache/beam/blob/6f31e56fcaca5cbb71b90f631c6562a1f1471c08/sdks/python/apache_beam/coders/coder_impl.py#L451
This can still be done but it is a bit more complicated and probably not
worth it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
