Ashfaqbs commented on issue #1084:
URL: https://github.com/apache/flink-agents/issues/1084#issuecomment-5683877747
I'd like to take this on, if unassigned. A rough direction, please correct
if it conflicts with what you had in mind:
The collision is specifically a same-action, same-seqNum, same-businessKey
fan-out where sibling events carry identical attributes (per `weiqingy`'s repro
on #950: a loop of `ctx.sendEvent(new Event("WorkItem", new HashMap<>(),
Map.of(...)))` gives every sibling the same empty `attributes`).
`RunnerContextImpl.sendEvent()` appends to an ordered `pendingEvents` list, and
siblings are drained/dispatched to the same action in that same order on both
the original run and any replay from the same source position -- so the Nth
time `ActionStateUtil.generateKey()` is called for an otherwise-identical
`(keyGroup, seqNum, eventUUID, actionUUID, businessKey)` tuple within a run is
itself a deterministic, replay-stable value.
Proposed approach: track an occurrence counter keyed by that same tuple
(minus the new ordinal), incremented each time `generateKey()` is called for
it, and append the ordinal as a new trailing segment before the business key
(keeping the business key last, per the existing parsing convention). The
counter only needs to live as long as the run that produced it -- it's a
source-order artifact, not something that needs its own durable persistence, as
long as replay re-derives it in the same order.
Two things I want to confirm before implementing, since this touches the
durable-state key format other stores parse against:
1. Is per-tuple call order actually guaranteed deterministic across replay
in every backend (Fluss/Kafka), or only under specific configurations?
2. Is a key-format change like this considered a breaking change for state
persisted under the current 5-segment format, or is pre-format state already
not preserved (per `ActionStateUtil.isKeyRetained`'s comment)?
Happy to write this up as a full proposal first if you'd rather review the
approach before I touch the key format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
