claudevdm commented on code in PR #36453:
URL: https://github.com/apache/beam/pull/36453#discussion_r2418008474
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -874,6 +874,18 @@ def _add_argparse_args(cls, parser):
'their condition met. Some operations, such as GroupByKey, disallow '
'this. This exists for cases where such loss is acceptable and for '
'backwards compatibility. See BEAM-9487.')
+ parser.add_argument(
+ '--force_cloudpickle_deterministic_coders',
Review Comment:
Actually as I was typing out the explanation I realized this flag wont work
in practice (because this change will result in a new coder name and therefore
failing update compat check in dataflow.
I added it so that tests that use update compat flags (like reshuffle tests)
that for some reason have a DeterministicFastPrimitivesCoder (but dont actually
trigger the pickle code path) fail because dill is not installed can be run in
non-dill test suites. These test have to be run in the dill test suite even
though they dont really need dill.
Maybe it should not be a flag/pipeline option, and just a property that we
hack on in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
