alisha-1000 opened a new issue, #41512:
URL: https://github.com/apache/beam/issues/41512

   ### What happened?
   
   **Summary**
   
   - With `@RequiresStableInput` on a stateful `DoFn` and the **heap** state 
backend, buffered elements and timers are replayed under the **wrong key** 
after a checkpoint completes, if two or more keys share a key group.
   - Effect: a timer set while processing an element of `k2` is registered 
under `k1`. With the same timer id, one key's timer overwrites the other's, and 
one timer output is lost.
   - Cause: `setCurrentKey` is called lazily inside a `flatMap`, but the stream 
is consumed through `.iterator()`, which buffers a whole key group first. All 
`setCurrentKey` calls for that group run before any element is processed, so 
every element runs under the last key of the group.
   
   Tested on master `5d834f2781b`, runners `flink:1.19` and `flink:2.2`, JDK 17.
   
   **Where**
   
   - `KeyedBufferingElementsHandler.getElements()` (lines ~67-81): calls 
`backend.setCurrentKey(key)` inside a `flatMap` over `getKeys(...)`.
   - On the heap backend, `getKeys(...)` is itself a `flatMap` over key groups.
   - `BufferingDoFnRunner` (lines ~307-316) consumes the stream with 
`.iterator()`. The JDK's iterator over a `flatMap` stream fills its buffer with 
the whole inner stream, so the `setCurrentKey` side effects for every key in 
the key group run before the first element is processed.
   
   **Reproduction**
   
   A `DoFnOperatorTest`-style test: keyed `@RequiresStableInput` `DoFn` that 
sets an event-time timer @50 for each key; elements for two keys, checkpoint, 
`notifyCheckpointComplete`, then watermark 100.
   
   | Backend | Keys | Result |
   | --- | --- | --- |
   | Heap, maxParallelism = 1 | k1, k2 (same key group) | **Fails**: only 
`KV{k1, timer}` |
   | Heap, maxParallelism = 128 | k1, k25 (both key group 51) | **Fails**: only 
one timer output |
   | Heap, maxParallelism = 128 | k1, k2 (different key groups) | Passes |
   
   Same results on flink:1.19 and flink:2.2. I also reproduced the ordering 
with a plain JDK program (`Stream.flatMap` + `.iterator()`): an element of `k1` 
is processed while the current key is `k2`.
   
   On RocksDB, `getKeys(...)` returns keys one at a time (no `flatMap`), and 
both timers fired in this test. RocksDB has a separate problem with buffered 
timer keys, which I'll file as a separate issue.
   
   **Impact**
   
   - Shown by tests: timers are registered under the wrong key, and a timer 
output is lost.
   - From reasoning, not tested: in a real job this happens whenever two keys 
with buffered data in the same checkpoint fall into the same key group, which 
becomes certain once a subtask buffers more distinct keys than it has key 
groups.
   - Not checked: whether user state (e.g. `ValueState`) is also written under 
the wrong key; behaviour on a real cluster; the portable runner.
   
   I have the reproducer test and can open a PR with it, or with a fix, if that 
helps.
   
   Found while working on #41448.
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [x] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam YAML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Infrastructure
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Prism Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to