sohailshk opened a new pull request, #41455:
URL: https://github.com/apache/beam/pull/41455
Fixes #21034
Since 2.29 the key coder of `GroupByKey`, `CombinePerKey`, `Distinct` and
stateful DoFns must be deterministic. For `datetime.date`, `datetime.datetime`
and `datetime.timedelta` keys this is `FastPrimitivesCoder`, which raises
"Unable to deterministically encode ... please provide a type hint". A type
hint does not help, because the registry maps these types to the same coder, so
`beam.Create([datetime.date(2020, 12, 2)] * 2).with_output_types(datetime.date)
| beam.Distinct()` fails as well. Today the only workarounds are converting the
keys or registering a custom coder.
```python
import datetime, apache_beam as beam
with beam.Pipeline() as p:
_ = (p | beam.Create([datetime.date(2020, 12, 2)] * 2) | beam.Distinct())
```
The deterministic encoder now writes these types field by field with three
new type tags: the ordinal for dates; the ordinal, microsecond of the day,
`fold` and a UTC flag for datetimes; and days, seconds and microseconds for
timedeltas. Only exact standard-library types are handled, and only naive or
`datetime.timezone.utc` datetimes, so decoding returns the same value with all
its fields. Other tzinfo objects and subclasses such as `pandas.Timestamp`
still raise. These values used to raise in deterministic mode, so no existing
encoding changes. The non-deterministic (pickle) path is untouched.
Tests: the new cases in `test_deterministic_coder` and
`test_distinct_on_dates` fail on master and pass with the change, with both the
pure-Python and the Cython-compiled `coder_impl`. `pytest apache_beam/coders`
(parallel and `no_xdist` runs): 165 passed (71 + 94). CHANGES.md updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]