alisha-1000 opened a new issue, #41512:
URL: https://github.com/apache/beam/issues/41512
### What happened?
**Summary**
- With `@RequiresStableInput` on a stateful `DoFn` and the **heap** state
backend, buffered elements and timers are replayed under the **wrong key**
after a checkpoint completes, if two or more keys share a key group.
- Effect: a timer set while processing an element of `k2` is registered
under `k1`. With the same timer id, one key's timer overwrites the other's, and
one timer output is lost.
- Cause: `setCurrentKey` is called lazily inside a `flatMap`, but the stream
is consumed through `.iterator()`, which buffers a whole key group first. All
`setCurrentKey` calls for that group run before any element is processed, so
every element runs under the last key of the group.
Tested on master `5d834f2781b`, runners `flink:1.19` and `flink:2.2`, JDK 17.
**Where**
- `KeyedBufferingElementsHandler.getElements()` (lines ~67-81): calls
`backend.setCurrentKey(key)` inside a `flatMap` over `getKeys(...)`.
- On the heap backend, `getKeys(...)` is itself a `flatMap` over key groups.
- `BufferingDoFnRunner` (lines ~307-316) consumes the stream with
`.iterator()`. The JDK's iterator over a `flatMap` stream fills its buffer with
the whole inner stream, so the `setCurrentKey` side effects for every key in
the key group run before the first element is processed.
**Reproduction**
A `DoFnOperatorTest`-style test: keyed `@RequiresStableInput` `DoFn` that
sets an event-time timer @50 for each key; elements for two keys, checkpoint,
`notifyCheckpointComplete`, then watermark 100.
| Backend | Keys | Result |
| --- | --- | --- |
| Heap, maxParallelism = 1 | k1, k2 (same key group) | **Fails**: only
`KV{k1, timer}` |
| Heap, maxParallelism = 128 | k1, k25 (both key group 51) | **Fails**: only
one timer output |
| Heap, maxParallelism = 128 | k1, k2 (different key groups) | Passes |
Same results on flink:1.19 and flink:2.2. I also reproduced the ordering
with a plain JDK program (`Stream.flatMap` + `.iterator()`): an element of `k1`
is processed while the current key is `k2`.
On RocksDB, `getKeys(...)` returns keys one at a time (no `flatMap`), and
both timers fired in this test. RocksDB has a separate problem with buffered
timer keys, which I'll file as a separate issue.
**Impact**
- Shown by tests: timers are registered under the wrong key, and a timer
output is lost.
- From reasoning, not tested: in a real job this happens whenever two keys
with buffered data in the same checkpoint fall into the same key group, which
becomes certain once a subtask buffers more distinct keys than it has key
groups.
- Not checked: whether user state (e.g. `ValueState`) is also written under
the wrong key; behaviour on a real cluster; the portable runner.
I have the reproducer test and can open a PR with it, or with a fix, if that
helps.
Found while working on #41448.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]