jk-shah opened a new issue, #41348:
URL: https://github.com/apache/beam/issues/41348

   ### What happened?
   
   When `Enrichment` (`RequestResponseIO`, 
`sdks/python/apache_beam/io/requestresponse.py`, lines `815-843`) runs with 
batching enabled (for example `BigQueryEnrichmentHandler`, where 
`min_batch_size=1, max_batch_size=10000` is enabled by default) on an 
event-time stream in `GlobalWindows` prior to `WindowInto`:
   
   1. `inputs | BatchElements(**self._batching_kwargs)` (line `816`) uses 
`_GlobalWindowsBatchingDoFn` (`transforms/util.py:808, 820`), which emits each 
batch at `GlobalWindows.windowed_value_at_end_of_window(self._batch)` 
(`Timestamp(9223371950454.774999)`).
   2. `responses | beam.ParDo(FlattenBatch())` (line `842`) yields 
un-timestamped elements from the batch, inheriting `END_OF_GLOBAL_WINDOW` 
(`Timestamp(9223371950454.774999)`).
   3. Any downstream `WindowInto(FixedWindows(...))` or timestamp-inspecting 
sink (`WriteToFiles` / `to_utc_datetime()`) assigns every record to year 
`294247` or fails with `OverflowError: date value out of range`.
   
   Minimal reproducer:
   
   ```python
   import apache_beam as beam
   from apache_beam.runners.portability import fn_api_runner
   from apache_beam.transforms.enrichment import Enrichment, 
EnrichmentSourceHandler
   from apache_beam.transforms.window import TimestampedValue
   
   class BatchedHandler(EnrichmentSourceHandler):
     def __call__(self, request, *args, **kwargs):
       return [(r, beam.Row(tier='gold')) for r in request]
     def batch_elements_kwargs(self):
       return {'min_batch_size': 1, 'max_batch_size': 10}
   
   with beam.Pipeline(runner=fn_api_runner.FnApiRunner()) as p:
     _ = (
         p
         | beam.Create([beam.Row(id=1), beam.Row(id=2)])
         | beam.Map(lambda r: TimestampedValue(r, 1700000005.0))
         | Enrichment(BatchedHandler())
         | beam.Map(lambda r, ts=beam.DoFn.TimestampParam: print('ts =', 
repr(ts)))
     )
   # Prints ts = Timestamp(9223371950454.774999) instead of 
Timestamp(1700000005)
   ```
   
   ### Cause
   
   Unlike general `BatchElements` (#22853, where input/output cardinality can 
be arbitrary), `RequestResponseIO` / `Enrichment` has a 1-to-1 correspondence 
between each input request and its `(request, response)` pair. Carrying 
`(element, timestamp, windows)` into the batch (with batch watermark hold at 
`min(timestamps)`) and restoring `WindowedValue` in `FlattenBatch` preserves 
exact per-element event-time timestamps.
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [x] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [x] Component: IO connector
   - [ ] Component: Beam YAML
   - [ ] Component: Beam ML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Infrastructure
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Prism Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to