AmoghAtreya opened a new pull request, #40765: URL: https://github.com/apache/beam/pull/40765
`WatchFilePattern` kept its newest-timestamp state under each file path. Beam partitions that state by key, so each path was compared only with the pipeline start time. Any file newer than startup was treated as the latest model, and a poll that saw several of them emitted all of them. `RunInference` reads this side input with `AsSingleton`, which rejects more than one value. Every watched file now shares one key, and the newest modification time seen so far is remembered. Each poll is reduced to the single newest file. A later file is emitted only when its modification time is newer than that. Files already present at pipeline construction stay on the model configured on the handler. A model update still needs a new filename. The file watcher reports a path once, and `RunInference` reloads only when the path changes. A rollback has to be uploaded under a new name with a newer modification time. Fixes #40638. ## Test plan - [x] `utils_test.py`: several files in one poll, out of order, emit only the newest model, and `AsSingleton` accepts that value. - [x] An older file that arrives in a later poll is not emitted. - [x] A newer file replaces an older one. - [x] Files at or before pipeline construction are not emitted. - [x] `CHANGES.md` notes the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
