GitHub user Ashfaqbs created a discussion: Design proposal for #1066: dry-run / replay via savepoints
Following up on #1066 since @wenjin272 asked for a concrete design before this goes anywhere. Putting a rough shape here to argue about, not committing to build it myself. ## The gap Right now the only way to know how a change to an agent (new prompt, different model, tweaked parameters, changed Action logic) will actually behave is to ship it and watch. There's no way to run it against state the agent has already built up and see what happens first. This is a decent fit for Flink Agents specifically because agent state is already durable and keyed, so a savepoint is a real, consistent snapshot to replay against - not something that needs to be built from scratch for this. ## Rough shape The core idea: a replay job is a regular Flink Agents job started from a savepoint, but reading from a recorded input stream instead of live traffic, with external side effects intercepted instead of fired. That breaks down into three parts. **Capturing the input.** An opt-in sink attached to the existing input stream that records the keyed events in order (event time, key, payload) so there's something deterministic to play back later. Doesn't touch runtime state - it just sits next to the existing pipeline. **Running the replay.** Flink already lets you start a job from a savepoint (`bin/flink run -s :savepointPath`). Point that at a `ReplaySource` reading the recorded stream instead of the real source, keeping per-key ordering and event-time so watermarks and keyed state behave the same as the original run. **Containing side effects.** This is the part I'd want the most pushback on. Actions that call out to a ChatModel or a tool need to behave differently in replay - either the call gets redirected somewhere inspectable instead of firing, or (if the whole point is testing a new prompt against real state) the call is allowed but anything not explicitly marked safe to repeat gets blocked. My instinct is a `ReplayContext` reachable from `RunnerContext`, defaulting to "don't execute, just record what would have happened" unless an Action opts in as idempotent/read-only. ## What it shouldn't need No changes to checkpoint or recovery semantics - from Flink's point of view a replay job is a normal job, just pointed at a different source. No new keyed state either; the capture sink and the savepoint restore both reuse what already exists. ## Where I'm not sure - Should side-effect safety be marked on the Action itself (a decorator) or kept in a separate registry? Decorator is more discoverable, but it ties a runtime concern to the Action definition. - Does the capture sink belong in core, or is it thin enough to just document as a pattern (a sink plus a source) without adding new API surface? - For a first cut, is comparing old-vs-new output on the same replayed input in scope, or is "run once against real state and look at it" enough to start with? No illusions about timing given the 0.4 freeze on Sept 15 - just want the shape settled for whenever it does get picked up. GitHub link: https://github.com/apache/flink-agents/discussions/1069 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
