mynameborat opened a new pull request, #1681: URL: https://github.com/apache/samza/pull/1681
**Description**: Currently, watermark is implemented as a special message within Samza. However, in terms of processing semantics, it shares similar behavior to normal messages processed by the task. i.e., `task.callback.timeout.ms`, a configuration to tune the time until which runloop waits for a message to be processed applies to both watermark and normal messages. However, this tie up constrains watermark processing logic to be bounded by the processing messages time bound. For Beam on Samza, we use watermark as a trigger to execute event timers which can take a long time depending on the number of timers accumulated. Especially, when the application is down, the timers accumulated could be too large and users will have to tune this configuration which will also impact fault tolerance behavior in case of failures/delays during processing messages. **Changes**: - Introduce callback timeout configuration specific to watermark - Update configuration documentation - Consolidate overload methods for `TaskCallbackManager` - Always use watermark specific timeout even when run loop is in draining mode **Tests**: - Added unit test for `TestRunLoop` - Added unit tests for `RunLoopConfig` **API Changes**: - Internal change to constructor **Upgrade Instructions**: None **Usage Instructions**: - Users can configure the timeout for watermark messages using `task.callback.watermark.timeout.ms` - Refer to the configuration documentation for more details and defaults. **Note**: Pull request is stacked on #1680 . Please review after 1st commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@samza.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
