Eliaaazzz opened a new pull request, #39352:
URL: https://github.com/apache/beam/pull/39352
This PR adds an experimental Python `Watch` transform and refactors the
deduplicated `FileIO.MatchContinuously` path to use it.
`Watch` polls each input with a user-provided `PollFn`, keeps only
never-seen-before outputs, advances output watermarks, and supports per-input
termination. The default deduplication behavior is by output key. A new
`use_timestamp` option lets callers include the output event timestamp in the
dedup key.
`MatchContinuously` now uses `Watch` when `has_deduplication=True`. Normal
deduplication is still by file path. When `match_updated_files=True`, the poll
function timestamps each `FileMetadata` output with `last_updated_in_seconds`,
and `Watch(use_timestamp=True)` emits the same path again when that timestamp
changes. The `has_deduplication=False` path remains `PeriodicImpulse` plus
`MatchAll` so every poll can re-emit all matches.
Validation:
- `python -m pytest -q sdks/python/apache_beam/io/watch_test.py
sdks/python/apache_beam/io/fileio_test.py::MatchContinuouslyTest`
- `git diff --check upstream/master..HEAD`
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description, if applicable. No
issue is currently linked.
- [ ] Update `CHANGES.md` with noteworthy changes. Not updated; this is an
experimental Python SDK transform/refactor.
- [ ] If this contribution is large, please file an Apache Individual
Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process
smoother.
To check the build health, please visit
https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
