sohailshk opened a new pull request, #41455:
URL: https://github.com/apache/beam/pull/41455

   Fixes #21034
   
   Since 2.29 the key coder of `GroupByKey`, `CombinePerKey`, `Distinct` and 
stateful DoFns must be deterministic. For `datetime.date`, `datetime.datetime` 
and `datetime.timedelta` keys this is `FastPrimitivesCoder`, which raises 
"Unable to deterministically encode ... please provide a type hint". A type 
hint does not help, because the registry maps these types to the same coder, so 
`beam.Create([datetime.date(2020, 12, 2)] * 2).with_output_types(datetime.date) 
| beam.Distinct()` fails as well. Today the only workarounds are converting the 
keys or registering a custom coder.
   
   ```python
   import datetime, apache_beam as beam
   with beam.Pipeline() as p:
     _ = (p | beam.Create([datetime.date(2020, 12, 2)] * 2) | beam.Distinct())
   ```
   
   The deterministic encoder now writes these types field by field with three 
new type tags: the ordinal for dates; the ordinal, microsecond of the day, 
`fold` and a UTC flag for datetimes; and days, seconds and microseconds for 
timedeltas. Only exact standard-library types are handled, and only naive or 
`datetime.timezone.utc` datetimes, so decoding returns the same value with all 
its fields. Other tzinfo objects and subclasses such as `pandas.Timestamp` 
still raise. These values used to raise in deterministic mode, so no existing 
encoding changes. The non-deterministic (pickle) path is untouched.
   
   Tests: the new cases in `test_deterministic_coder` and 
`test_distinct_on_dates` fail on master and pass with the change, with both the 
pure-Python and the Cython-compiled `coder_impl`. `pytest apache_beam/coders` 
(parallel and `no_xdist` runs): 165 passed (71 + 94). CHANGES.md updated.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to