robertwb commented on a change in pull request #10972: [WIP] Add
DeduplicationByUniqueId transform
URL: https://github.com/apache/beam/pull/10972#discussion_r385429733
##########
File path: sdks/python/apache_beam/runners/sdf_utils.py
##########
@@ -244,3 +247,47 @@ def get_estimator_state(self):
return None
return _NoOpWatermarkEstimator()
+
+
+class DeduplicationByUniqueId(ptransform.PTransform):
+ def __init__(self, windowfn=GlobalWindows(), trigger=None):
Review comment:
I think if the user wants to specify things in terms of windows and
triggers, it would be more natural to manually do windowing before this
operation. Instead, perhaps higher-level semantic information could be provided
(namely, over what interval should the deduplication occur), and
windowing/triggering should be used to accomplish this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
