tkaymak opened a new issue, #40517: URL: https://github.com/apache/beam/issues/40517
### What needs to happen? `BeamStatefulProcessor` keeps a `beamWatermark` and a `beamTimers` `ValueState` per grouping key and never clears `beamWatermark`, so a stateful ParDo, and the upcoming windowed GroupByKey, over an unbounded key space grows the state store forever, even after all windows of a key expired and its `beamState` map is empty. Clear both when no timers remain and `beamState` is empty, or attach a `TTLConfig` derived from allowed lateness. Part of #36841, follows #40447. ### Issue Priority Priority: 2 (default / most feature requests should be filed as P2) ### Issue Components - [x] Component: Spark Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
