github-actions[bot] commented on code in PR #41370:
URL: https://github.com/apache/beam/pull/41370#discussion_r4579180280


##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -1274,6 +1274,83 @@ def __init__(self, num_buckets: Optional[int] = None):
     def expand(self, pcoll):
       return pcoll | beam.Reshuffle(num_buckets=self.num_buckets)
 
+  class Deduplicate(beam.PTransform):
+    """Drops records whose key fields have already been seen recently.
+
+    Only the first record with a given set of key field values is kept, and
+    any other record with the same values that arrives within `duration` of it
+    is dropped. Records in different windows are never considered duplicates
+    of each other. The output has the same schema and timestamps as the input.
+
+    For example::
+
+        type: Deduplicate
+        config:
+          keys: [tenant_id, event_id]
+          duration: 10m
+          time_domain: processing_time
+
+    The duration must be defined using a time unit suffix 'ms', 's', 'm', 'h'
+    or 'd' for milliseconds, seconds, minutes, hours or days, respectively.
+    If a time unit is not specified, it will default to 's'.
+
+    The number of kept and dropped records are reported in the
+    `deduplicate_kept` and `deduplicate_dropped` counters.
+
+    Args:
+      keys: The field(s) whose values identify duplicate records.
+      duration: How long a key is remembered after the first record with it.
+      time_domain: The time domain `duration` is measured in, either
+        `processing_time` (the default) or `event_time`.
+    """
+    def __init__(
+        self,
+        keys: Iterable[str],
+        duration: str,
+        time_domain: Optional[str] = 'processing_time'):
+      if isinstance(keys, str):
+        keys = [keys]
+      if not keys:
+        raise ValueError('Deduplicate requires at least one key field.')
+      self._keys = list(keys)

Review Comment:
   **🟢 Low**: If `keys` contains duplicate field names (e.g., `keys: 
[tenant_id, tenant_id]`), `RowTypeConstraint.from_fields` will construct a 
schema with duplicate field names while `beam.Row(**{k: getattr(row, k) for k 
in keys})` collapses them into a single keyword argument, causing `RowCoder` 
(`named_tuple_from_schema`) to fail with `ValueError: Encountered duplicate 
field name`. Additionally, if a non-sequence `Iterable[str]` (such as a 
generator) is passed from Python, `if not keys:` above will evaluate truthy 
before `list(keys)` is materialized. Deduplicating with `dict.fromkeys(keys)` 
(and checking `if not self._keys:` after materialization) avoids both edge 
cases while preserving key order.
   
   ```suggestion
         self._keys = list(dict.fromkeys(keys))
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to