jk-shah opened a new issue, #41348:
URL: https://github.com/apache/beam/issues/41348
### What happened?
When `Enrichment` (`RequestResponseIO`,
`sdks/python/apache_beam/io/requestresponse.py`, lines `815-843`) runs with
batching enabled (for example `BigQueryEnrichmentHandler`, where
`min_batch_size=1, max_batch_size=10000` is enabled by default) on an
event-time stream in `GlobalWindows` prior to `WindowInto`:
1. `inputs | BatchElements(**self._batching_kwargs)` (line `816`) uses
`_GlobalWindowsBatchingDoFn` (`transforms/util.py:808, 820`), which emits each
batch at `GlobalWindows.windowed_value_at_end_of_window(self._batch)`
(`Timestamp(9223371950454.774999)`).
2. `responses | beam.ParDo(FlattenBatch())` (line `842`) yields
un-timestamped elements from the batch, inheriting `END_OF_GLOBAL_WINDOW`
(`Timestamp(9223371950454.774999)`).
3. Any downstream `WindowInto(FixedWindows(...))` or timestamp-inspecting
sink (`WriteToFiles` / `to_utc_datetime()`) assigns every record to year
`294247` or fails with `OverflowError: date value out of range`.
Minimal reproducer:
```python
import apache_beam as beam
from apache_beam.runners.portability import fn_api_runner
from apache_beam.transforms.enrichment import Enrichment,
EnrichmentSourceHandler
from apache_beam.transforms.window import TimestampedValue
class BatchedHandler(EnrichmentSourceHandler):
def __call__(self, request, *args, **kwargs):
return [(r, beam.Row(tier='gold')) for r in request]
def batch_elements_kwargs(self):
return {'min_batch_size': 1, 'max_batch_size': 10}
with beam.Pipeline(runner=fn_api_runner.FnApiRunner()) as p:
_ = (
p
| beam.Create([beam.Row(id=1), beam.Row(id=2)])
| beam.Map(lambda r: TimestampedValue(r, 1700000005.0))
| Enrichment(BatchedHandler())
| beam.Map(lambda r, ts=beam.DoFn.TimestampParam: print('ts =',
repr(ts)))
)
# Prints ts = Timestamp(9223371950454.774999) instead of
Timestamp(1700000005)
```
### Cause
Unlike general `BatchElements` (#22853, where input/output cardinality can
be arbitrary), `RequestResponseIO` / `Enrichment` has a 1-to-1 correspondence
between each input request and its `(request, response)` pair. Carrying
`(element, timestamp, windows)` into the batch (with batch watermark hold at
`min(timestamps)`) and restoring `WindowedValue` in `FlattenBatch` preserves
exact per-element event-time timestamps.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam ML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]