[
https://issues.apache.org/jira/browse/BEAM-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476454#comment-17476454
]
Valentyn Tymofieiev commented on BEAM-12717:
--------------------------------------------
https://issues.apache.org/jira/browse/BEAM-11719 started to enforced
deterministic coders for GBK.
Release notes for https://beam.apache.org/blog/beam-2.29.0/ mention this as a
breaking change and say:
Deterministic coding enforced for GroupByKey and Stateful DoFns. Previously
non-deterministic coding was allowed, resulting in keys not properly being
grouped in some cases. (BEAM-11719) To restore the old behavior, one can
register FakeDeterministicFastPrimitivesCoder with
beam.coders.registry.register_fallback_coder(beam.coders.coders.FakeDeterministicFastPrimitivesCoder())
or use the allow_non_deterministic_key_coders pipeline option.
[~robertwb] - do users also have an option to define and register a custom
deterministic coder for the datatypes involved or that would need to be done in
the SDK itself? Also, is the error message "please provide a type hint for the
input of..." useful here? That message seems to be from some older code,
perhaps we should amend it due to this change.
> datetime.date not working with Distinct()
> -----------------------------------------
>
> Key: BEAM-12717
> URL: https://issues.apache.org/jira/browse/BEAM-12717
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.31.0
> Reporter: Sebastian Kretzig
> Priority: P3
> Labels: types
>
> I was using 2.28.0 and running a standard Distinct on a PColl of
> datetime.dates without problems.
> After upgrading to 2.31.0, I'm getting this error:
> TypeError: Unable to deterministically encode '2020-12-02' of type '<class
> 'datetime.date'>', please provide a type hint for the input of 'Get the
> unique dates from prices/Group/GroupByKey' [while running 'Get the unique
> dates from prices/ToPairs']
> Had convert the date to str and back to date after running the Distinct() to
> preserve the date type. Any clues?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
