Gowtham-Gowts opened a new pull request, #8787: URL: https://github.com/apache/storm/pull/8787
## Summary Adds `topology.message.progress.timeout.secs` — an opt-in config that changes tuple expiry from wall-clock-since-emit to time-since-last-bolt-progress. When set, a tuple tree is only expired if no bolt has acked any tuple in the tree for N seconds. Tuple trees actively being processed (even if queued under backpressure) are rescued from RotatingMap rotation as long as forward progress is being made. `topology.message.timeout.secs` remains the hard upper-bound, unchanged. ## Motivation Raised on dev@ (June 2026): https://lists.apache.org/thread/tjcso52j1tjk6d344vjfmkd9ngy6k2rb Users face an inherent tradeoff: - Short timeout → orphans reclaimed quickly, but live tuples fail under backpressure - Long timeout → backpressure-safe, but orphan reclamation is slow after worker failure This config dissolves that tradeoff. Discussed with rzo1 on the issue tracker: https://github.com/apache/storm/issues/6141 ## Changes | File | Change | |---|---| | `Config.java` | New `TOPOLOGY_MESSAGE_PROGRESS_TIMEOUT_SECS` constant | | `Acker.java` | `lastProgressTime` in `AckObject`; `rescueRecentlyActiveEntries()` | | `RotatingMap.java` | `peekOldestBucket()` package-private method | | `Executor.java` | WARN log for STORM-3514 dangerous config combo | | `AckerProgressTimeoutTest.java` | 5 new unit tests | | `RotatingMapPeekTest.java` | 6 new unit tests | | `conf/defaults.yaml` | null default entry with comment | | `docs/Guaranteeing-message-processing.md` | New section | ## Backward Compatibility - **Off by default** — `topology.message.progress.timeout.secs` is `null`. Zero behavior change for all existing topologies. - No Thrift IDL changes - No new stream IDs - No bolt or spout API changes - Memory cost: +8 bytes per in-flight tuple tree (one `long` field) ## Related Issues - STORM-2359: https://github.com/apache/storm/issues/6141 - STORM-3314: https://github.com/apache/storm/issues/7096 - STORM-3514: https://github.com/apache/storm/issues/7296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
