schizophrenicmaniac opened a new pull request, #41370:
URL: https://github.com/apache/beam/pull/41370
Adds a `Deduplicate` transform to Beam YAML so pipelines can drop repeated
deliveries by one or more fields without leaving YAML:
```yaml
- type: Deduplicate
config:
keys: [tenant_id, event_id]
duration: 10m
time_domain: processing_time # or event_time
```
Each Row is keyed by the given fields and passed through the existing
`DeduplicatePerKey`, so `duration` and `time_domain` map straight onto its
processing/event time durations. The key is a schema'd Row, which gives a
deterministic RowCoder for the stateful step, and null key values work. The
original Row is emitted unchanged, so the output keeps the input schema and
timestamps. `duration` uses the same parsing as `WindowInto` (`ms`, `s`, `m`,
`h`, `d`). Like `DeduplicatePerKey`, records in different windows are not
treated as duplicates.
I also added `deduplicate_kept` and `deduplicate_dropped` counters to
`DeduplicatePerKey`. The YAML wrapper can't see dropped records, so the
counters have to live there, and Python users of `DeduplicatePerKey` get them
as well.
Tests are in the new `yaml_deduplicate_test.py` (single/composite/nullable
keys, event time, schema and timestamps kept, separate windows, counters,
config errors), plus a `tests/deduplicate.yaml` integration case. CHANGES.md is
updated.
Fixes #41354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]