claudevdm commented on code in PR #39943:
URL: https://github.com/apache/beam/pull/39943#discussion_r4017241636
##########
sdks/python/apache_beam/coders/coders.py:
##########
@@ -914,7 +915,8 @@ def _create_impl(self):
lambda x: dumps(x, protocol), pickle.loads)
def as_deterministic_coder(self, step_label, error_message=None):
- return FastPrimitivesCoder(self, requires_deterministic=step_label)
+ return _update_compatible_deterministic_fast_primitives_coder(
Review Comment:
Returning DeterministicFastPrimitivesCoder will fail at runtime for
arbitrary classes.
https://github.com/apache/beam/blob/42663f32154b454c8bc64dd312373b3138c84f1c/sdks/python/apache_beam/coders/coder_impl.py#L486
Maybe add a warning like
```
PickleCoder was registered for a key type in '%s', but the runner requires a
deterministic key encoding and pickle is not deterministic. Keys in this step
will be encoded with the deterministic fallback coder instead of pickle. That
coder supports primitives, containers, protobuf messages, frozen dataclasses,
NamedTuples, enums, and classes defining both __getstate__ and __setstate__.
Any other key type will fail at encode time. If your key type is not one of
these, register a deterministic custom Coder for it or add a type hint so the
default coder is used.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
