Abacn commented on code in PR #39461:
URL: https://github.com/apache/beam/pull/39461#discussion_r3787976147
##########
sdks/python/apache_beam/io/fileio.py:
##########
@@ -299,44 +445,93 @@ def __init__(
self.match_upd = match_updated_files
self.apply_windowing = apply_windowing
self.empty_match_treatment = empty_match_treatment
- _LOGGER.warning(
- 'Matching Continuously is stateful, and can scale poorly. '
- 'Consider using Pub/Sub Notifications '
- '(https://cloud.google.com/storage/docs/pubsub-notifications) '
- 'if possible')
+ self.timestamp_cursor = timestamp_cursor
+ if timestamp_cursor and not has_deduplication:
+ raise ValueError(
+ 'MatchContinuously(timestamp_cursor=True) deduplicates, so it '
+ 'requires has_deduplication=True.')
Review Comment:
```suggestion
if timestamp_cursor:
if not has_deduplication:
raise ValueError(
'MatchContinuously(timestamp_cursor=True) deduplicates, so it '
'requires has_deduplication=True.')
if not match_updated_files:
_LOGGER.warning("timestamp_cursor implies match_upd, set
match_upd=True")
self.match_upd = True
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
