schizophrenicmaniac opened a new pull request, #41370:
URL: https://github.com/apache/beam/pull/41370

   Adds a `Deduplicate` transform to Beam YAML so pipelines can drop repeated 
deliveries by one or more fields without leaving YAML:
   
   ```yaml
   - type: Deduplicate
     config:
       keys: [tenant_id, event_id]
       duration: 10m
       time_domain: processing_time   # or event_time
   ```
   
   Each Row is keyed by the given fields and passed through the existing 
`DeduplicatePerKey`, so `duration` and `time_domain` map straight onto its 
processing/event time durations. The key is a schema'd Row, which gives a 
deterministic RowCoder for the stateful step, and null key values work. The 
original Row is emitted unchanged, so the output keeps the input schema and 
timestamps. `duration` uses the same parsing as `WindowInto` (`ms`, `s`, `m`, 
`h`, `d`). Like `DeduplicatePerKey`, records in different windows are not 
treated as duplicates.
   
   I also added `deduplicate_kept` and `deduplicate_dropped` counters to 
`DeduplicatePerKey`. The YAML wrapper can't see dropped records, so the 
counters have to live there, and Python users of `DeduplicatePerKey` get them 
as well.
   
   Tests are in the new `yaml_deduplicate_test.py` (single/composite/nullable 
keys, event time, schema and timestamps kept, separate windows, counters, 
config errors), plus a `tests/deduplicate.yaml` integration case. CHANGES.md is 
updated.
   
   Fixes #41354


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to