alisha-1000 opened a new issue, #41513: URL: https://github.com/apache/beam/issues/41513
### What happened? **Summary** - With `@RequiresStableInput` on a stateful `DoFn`, a timer that fires and is buffered until the checkpoint completes is replayed with a **null key**. - Effect: `@Key` in `@OnTimer` is `null` for the replayed timer. - Cause: `BufferedElements.Coder` doesn't encode the key of a buffered timer. On decode it uses the coder's own key, and `BufferingDoFnRunner` creates that coder with a `null` key. Tested on master `5d834f2781b`, runner `flink:1.19`, JDK 17. **Where** - `BufferedElements.Coder.encode` (`BufferedElements.java`, lines ~164-172): writes the timer's id, family, window, timestamps and time domain, but not its key. - `BufferedElements.Coder.decode` (line ~188): rebuilds the timer with the coder's own key. - `BufferingDoFnRunner` (line ~184): constructs the coder with a `null` key. **When it happens** The key is lost whenever a buffered timer goes through the coder: | Backend | Restored from checkpoint | Key in `@OnTimer` | | --- | --- | --- | | RocksDB | no | `null` | | RocksDB | yes | `null` | | Heap | yes | `null` | | Heap | no | correct | RocksDB always serializes state, so it hits this on every replay. The heap backend keeps the object in memory, so it only hits this after a restore. **Reproduction** A `DoFnOperatorTest`-style test: keyed `@RequiresStableInput` `DoFn` with an event-time timer whose `@OnTimer` method reads `@Key`. Process an element, complete a checkpoint, advance the watermark so the timer fires and is buffered, then complete the next checkpoint (optionally restoring from a snapshot in between). The timer fires once, but `@Key` is `null`. **Impact** - Shown by tests: `@Key` in `@OnTimer` is `null` for buffered timers, on RocksDB always and on heap after restore. - From reading the code, not tested: `@OnWindowExpiration` likely gets a `null` key too. - Not checked: whether user code then fails (e.g. NPE or an encoding error); behaviour on a real cluster; the portable runner. Related: #41512 (heap backend, wrong key context on replay), found at the same time. I have the reproducer test and can open a PR with it or a fix. Found while working on #41448. ### Issue Priority Priority: 2 (default / most bugs should be filed as P2) ### Issue Components - [ ] Component: Python SDK - [ ] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [ ] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Infrastructure - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Prism Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
