GitHub user sangkyoonnam added a comment to the discussion: Planning Flink Agents 1.0 - Brainstorming
I ran two cross-release experiments with the operator test harness: Java actions, HashMap backend, no Action State Store. For 0.3.0 to 0.4.0-rc1, snapshots taken while an action was pending failed to restore with a `KryoException`. The traces name `Action.triggerConditions` and `Event.attachments`, both of which changed in 0.4, though I haven't isolated which change causes it. For 0.2.1 to 0.3.0, a snapshot with the second action pending failed with a `NullPointerException` while reading keyed state. Snapshots with no pending action restored in both cases, and same-version and Flink-only (2.2.1 to 2.3.0) controls passed. I haven't tried Python actions, RocksDB or a cluster savepoint. `actionTasks` and `pendingInputEvents` use Kryo for `ActionTask` and `Event`, and Flink's `KryoSerializerSnapshot` checks serializer configuration and registrations rather than field layouts, so the compatibility check can miss field changes in those classes. `MemoryItem` and the `Object`-typed union lists (`currentProcessingKeys`, `recoveryMarker`) use Kryo as well; these runs didn't exercise `recoveryMarker`. I opened #1332 for test infrastructure and an at-rest scenario, with no production code changes. Three decisions shape what follows: - Should the guarantee cover only upgrades within 1.x, or 0.4 to 1.0 as well? That decides whether serializer changes before 1.0 need a way to read 0.4 state. - Must upgrades work with in-flight work in the snapshot, or may they require draining first? If in-flight work is supported, I'd lean toward rerunning unfinished actions while reusing durably recorded results, which still needs readable pending inputs, stable action identities and a cross-version journal format. - Should the agent API let users set the operator uid before 1.0? `CompileUtils.connectToAgent` sets a name but no uid, and `AgentBuilder.toDataStream()` is declared to return `DataStream`, which doesn't expose `uid()`. GitHub link: https://github.com/apache/flink-agents/discussions/1327#discussioncomment-19231849 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
