Hi all,
Thanks for looping me in, André, and thanks for putting this together. It's
a really nice piece of work. The suspend/resume flow is clean and the
DeterminismError guard is a thoughtful touch. I left some detailed notes on
the PR, but since the design conversation is here, let me share one
observation from testing plus where I land on the open questions.
One thing from testing the branch that might be worth a look (crash
recovery).
The journal currently gets reloaded inside resume(), but not on the normal
recovery path (initialize_from(..., resume_at_next_action=True)). So if a
process dies mid-action without a suspension, the memoized sub-steps seem
to re-run on restart rather than replay. I put together a quick repro to be
sure I wasn't misreading it: a two-action graph, crash the second action
after its first durable() call had journaled, then recover in a fresh
process:
loaded_journal on recovery: []
manual load_journal(pk, app, seq): [('step_a', 'A')] # the row is there,
key matches
step_a re-fired on recovery? True
The journal row is written and the key lines up (I double-checked the
sequence_id), it just isn't loaded outside resume(). The PR's exactly-once
test passes because it goes through resume(), which is the path that loads
it. I mention it only because it touches the case 2 (crash recovery)
framing. If the intended scope is deliberate suspend/resume, this is
totally fine and just a docs note; if we want crashes covered too, loading
the journal on the recovery path looks like a small change. Curious what
you all see as the intended scope.
On André's Q1 (halt_when as a primitive vs conditional edges): I lean
toward Stefan's instinct here. Keeping it expressed as state plus
conditional edges feels like less new surface area, and
halt_when(durable=[...]) as thin sugar over that, the way Elijah suggested,
seems like a nice middle ground that doesn't introduce a brand-new
transition concept.
On Q2 (the durable() API surface): both forms read well to me. The callable
form that's in the PR is the easiest to onboard and to unit-test, which is
worth something. Elijah's Azure-style yield ctx.durable(...) with match is
elegant and I can see the appeal, my only small hesitation is that turning
actions into generators changes the mental model a bit. If it's easy,
supporting the callable form first and revisiting the generator form once
people are using it might keep the initial surface small. Just a
preference, not a strong view.
On Elijah's "can we just model it as sub-state" question: I think that
direction is reasonable, and the in-state path mostly already does it. The
one thing I noticed in testing is that the in-state journal isn't cleared
after an action completes, so it accumulates across the run and the blob
gets rewritten each step. That seems fixable within the in-state model
(clear it on completion, keep only what the suspended action needs), which
to me actually supports Elijah's "keep it in state" instinct rather than
arguing for a separate tier.
On the two-tier / resume-once point: I'd also lean toward not solving
resume concurrency in this PR, since Burr doesn't have a resume concurrency
model elsewhere. One small factual note for the record: the atomic
primitive is already present per-persister (mark_suspension_resolved is a
conditional update returning a bool), but the resume path reads
record.resolved before the run and marks it after, so it isn't gating on
that result yet. Happy to leave that out of scope for now and just note in
the docs that the current path is effectively single-writer.
On hierarchy: just confirming what you already flagged, André, the journal
key is (partition_key, app_id, sequence_id, step_key) today with no parent
app id, so your plan to fold the parent app id in matches what's there.
On Q3 (anyone hitting the mid-action sequential case in production): I'm
not running this exact pattern in production myself, but the underlying
cost (re-paying for an LLM batch on retry) is very familiar from data/ML
pipeline work, so the motivation resonates.
Stefan, on the gist idea, I'd be glad to help here . I can write up the
loyalty-CRM batch-with-approval flow against the current branch and include
the crash repro above, so we have something concrete to talk UX around.
Happy to start it if it's useful.
Thanks again for driving this, it's a genuinely valuable feature and I'm
glad to help however works best.
Regards,
Viquar Khan
https://www.linkedin.com/in/vaquar-khan-b695577/
On Sun, Jun 14, 2026 at 10:35 PM Elijah ben Izzy <
[email protected]> wrote:
> Not sure I agree with claude's points here will think it over:
> 1. *State blob growth.* -- This depends on assumptions about modeling it
> that claude made -- I think that if we model it closer to state it'll be
> the same.
> 2. *Resume-once under concurrency.* This is a whole can of worms we
> don't really support a resume concurrency model anywhere? Why would this be
> special? We'd need to get a lock etc...
> 3. *Queryability.* Sure could be a boolean flag that is by default false
> 4. *Schema coupling.* The state could be "waiting" effectively -- it
> *is* state...
>
> not saying I disagree with the approach necessarily but worth thinking
> through
>
> On Sun, Jun 7, 2026 at 3:10 PM Stefan Krawczyk <[email protected]>
> wrote:
>
> > Can we start some gists? Or draft PRs? That would help me if we talked UX
> > and what it would mean. E.g. let's take that loyalty CRM case and look at
> > the API one would use to implement it.
> >
> > On Mon, Jun 1, 2026, 12:41 PM André Ahlert <[email protected]> wrote:
> >
> > > Thanks Elijah, this is the right question to push on. Let me answer it
> > head
> > > on, because the answer is "yes there is an easier way, and I think we
> > > should ship it as the default."
> > >
> > > You are right that a durable state can ride inside the existing state
> > model
> > > rather than a parallel structure. PR #786 already proves this: custom
> > > persisters fall back to an in-state journal, and that path is correct.
> No
> > > new persister methods, no dedicated tables, onboarding is zero. For a
> > > single-worker workflow it is the simplest thing that works, and I do
> not
> > > want to force anyone onto a heavier model to get suspend/resume.
> > >
> > > So I want to reframe the design as two tiers rather than one monolith:
> > >
> > > - *Default (light):* journal and suspension records live in the
> > existing
> > > state keyspace, exactly as you suggested. Any persister gets durable
> > > execution for free. This is the onboarding path.
> > > - *Opt-in (heavy):* a persister that implements the dedicated
> methods
> > (
> > > save_suspension, mark_suspension_resolved, etc.) for workloads that
> > > outgrow the light path.
> > >
> > > The reason the opt-in tier earns its keep is not aesthetics, it is four
> > > concrete costs the in-state path hits at scale:
> > >
> > > 1. *State blob growth.* Journal entries and suspension payloads
> inside
> > > state mean every save reserializes the whole blob. A workflow with N
> > > memoized sub-steps rewrites O(N) state on every step. Fine for small
> > > flows,
> > > painful for long ones.
> > > 2. *Resume-once under concurrency.* Two workers resuming the same
> > > suspension need an atomic claim. The SQL persisters expose exactly
> > that
> > > primitive: mark_suspension_resolved is a conditional UPDATE ...
> WHERE
> > > resolved = 0 returning rowcount > 0, so under concurrency exactly
> one
> > > caller wins. In-state cannot express that primitive at all. To be
> > > precise
> > > about the current state of the branch: resume() / aresume() still
> > gate
> > > on a non-atomic record.resolved read taken before the run and
> discard
> > > the bool, so the race is not yet closed end to end. Wiring
> resumption
> > to
> > > skip the run unless mark_suspension_resolved returns True is part of
> > > this work, and it is only expressible on the dedicated tier (the
> > > in-memory
> > > persister is a plain loop, not atomic).
> > > 3. *Queryability.* "List all pending suspensions awaiting approval"
> is
> > > one SQL query against a table. In-state it is a full scan of every
> > app's
> > > serialized state. Not viable for an ops dashboard or the Burr UI.
> > > 4. *Schema coupling.* Mixing runtime metadata into the user state
> > > keyspace invites key collisions and muddies "what is my state"
> versus
> > > "what
> > > is runtime bookkeeping."
> > >
> > > *The real case driving this.* A loyalty CRM I am running in
> production. A
> > > campaign action generates a personalized offer per customer via an LLM,
> > > processed as a sequential batch of ~50. Two things break today:
> > >
> > > - Item 23 needs human approval before the offer goes out. Items 1
> > > through 22 have already paid for their LLM calls. With halt_before/
> > > halt_after I cannot pause mid batch, so I either fork state manually
> > or
> > > re-run the batch and pay for 1 through 22 again.
> > > - On any crash or retry mid batch, every offer regenerates. The
> > > offer-generation LLM is the dominant cost in the workflow, so a
> single
> > > retry doubles the spend for that run.
> > >
> > > With __context.durable("offer", generate_offer, customer) the replay
> > reads
> > > the recorded offers and re-fires nothing. This is test-backed in the
> PR:
> > > the integration suite asserts a durable side effect runs exactly once
> > > across a suspend/resume cycle (the recorded value is replayed, the
> > function
> > > is not re-invoked). I also ran the HITL example end to end against a
> > local
> > > model to confirm the same behavior outside the test harness.
> > >
> > > Now, here is where the in-state default would actually hurt this case,
> > > which is why I want the opt-in tier available: the recorded offer
> > payloads
> > > sit in the state blob and get reserialized on every subsequent step,
> and
> > > the approval step is exactly where a second worker could pick up the
> same
> > > suspension. Costs 1 and 2 are not hypothetical for this workload.
> > >
> > > *On the API surface.* I like your Azure-style yield ctx.durable(...)
> with
> > > match. It reads cleanly. The current PR uses the callable form
> > > ctx.durable(key,
> > > fn, ...); I am open to the generator form if folks prefer it, the
> journal
> > > semantics are the same underneath. And halt_when(durable=[...]) as thin
> > > sugar over conditional edges is a nice touch, that also answers
> Stefan's
> > > preference to keep halting expressed as state plus edges rather than a
> > new
> > > transition primitive.
> > >
> > > *On Stefan's hierarchy point:* the journal key is (partition_key,
> app_id,
> > > sequence_id, step_key) today, with entries ordered by call_index. It
> does
> > > not yet fold in a parent app id, so nested apps do not inherit a parent
> > > journal. I will make the key compose with the parent app id so nested
> > apps
> > > inherit the journal correctly, which is the hierarchical reuse you
> > flagged.
> > >
> > > Proposed path: keep suspend/resume and the journal in #786, make the
> > > in-state path the documented default, mark the dedicated persister
> > methods
> > > as the opt-in scale tier, and address hierarchy in the same PR. If that
> > > shape works for both of you I will split out anything that does not
> need
> > to
> > > land together.
> > >
> > > No deadline. Will send a [PROPOSAL] once we converge.
> > >
> > > PS: sorry for the blogpost in dev list, but is a big theme. :)
> > >
> > > André
> > >
> > > Em dom., 31 de mai. de 2026 às 22:58, Elijah ben Izzy <
> > > [email protected]> escreveu:
> > >
> > > > This is very cool. So I thought through there's definitely a need
> here
> > to
> > > > pause mid-action.
> > > >
> > > > Some API ideas:
> > > > 1. Put durable_keys as part of the declaration to match the
> type-safe,
> > > > declarative nature of the rest of it
> > > > 2. Use something clever (i.e. match statements) to make it easier to
> > > match
> > > > the mental model
> > > >
> > > > Azure durable functions has an interesting approach:
> > > >
> > > > def my_action(state, ctx):
> > > > user = yield ctx.durable("fetch", db.get_user, state["uid"])
> > > > match user.tier:
> > > > case "gold":
> > > > result = yield ctx.durable("gold_flow", gold_flow,
> user)
> > > > case _:
> > > > result = yield ctx.durable("std_flow", std_flow, user)
> > > > return state.update(result=result)
> > > >
> > > > Worth exploring. I'm also wondering if there's a simple way to add an
> > > > optional field to the data model rather than a whole new "supports
> > > > durable"? I.E. would there be an easier way to onboard? Could we
> model
> > it
> > > > as some sort of sub-state that we could store and pass in? If we had
> > > > durable things as a special key-space in the state could we just use
> > the
> > > > same ones? Trade-offs not sure the right one.
> > > >
> > > > The problem is that if we don't have suspend() we decouple it but
> need
> > a
> > > > mechanism to halt execution. Might be a good place for syntactic
> sugar
> > --
> > > > halt_during? halt_when(durable=["key"])?
> > > >
> > > > With iterator-based APIs it's actually up to the user to handle the
> > > halting
> > > > -- this completely bypasses this issue (we just send out a
> durable-set
> > > > event mid-stride as we would an iterator or an iterator of streams).
> > > >
> > > >
> > > >
> > > > On Sun, May 31, 2026 at 11:45 AM Stefan Krawczyk <
> > > > [email protected]>
> > > > wrote:
> > > >
> > > > > If halt_when has clear semantics if what happens next then that
> > sounds
> > > > > good. But I haven't found a clear way to make that obvious. So I
> > think
> > > > > making users set up state and conditional edges should be the
> cleaner
> > > > model
> > > > > (we should also show that running a graph with halt* is fine in
> > > examples
> > > > > and drop that warning).
> > > > >
> > > > > Durable key seems fine I think. I think the hierarchical nature
> > should
> > > be
> > > > > utilized that we've set up, so more capabilities to enable that way
> > of
> > > > > pausing and resuming to be more useful makes sense to me -- IIUC.
> > > > >
> > > > > On Thu, May 28, 2026, 6:14 AM André Ahlert <[email protected]>
> > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > PR #786 <https://github.com/apache/burr/pull/786> [1] proposes
> two
> > > > > > additions to handle long-running agent workloads. Moving the
> design
> > > > > > question from DM with Stefan onto the list before we iterate
> > further.
> > > > > >
> > > > > > Gap today: halt_before / halt_after only stop between actions.
> Two
> > > > cases
> > > > > > keep showing up:
> > > > > >
> > > > > > 1. HITL inside one action (e.g. sequential batch where item 23
> > of
> > > 50
> > > > > > needs approval, items 1-22 already paid for LLM calls).
> > > > > > 2. Crash recovery mid-action without re-firing paid sub-steps.
> > > > > >
> > > > > > Langgraph covers both with interrupt() + checkpointer. Concrete
> > case
> > > I
> > > > am
> > > > > > hitting: a loyalty CRM where every retry re-fires the
> > > offer-generation
> > > > > LLM.
> > > > > >
> > > > > > Proposal is to split #786 into two orthogonal pieces, ship
> > > > independently:
> > > > > >
> > > > > > * A. *halt_when(predicate): state-based HITL primitive. Action
> > sets a
> > > > > state
> > > > > > value, runtime halts when predicate matches, conditional edges
> > route
> > > on
> > > > > > resume. No new resume semantics. (Stefan's suggested shape on the
> > > PR.)
> > > > > >
> > > > > > * B. *__context.durable(key, fn): sub-step memoization journal.
> > > Result
> > > > > > keyed by (app_id, sequence, key). On replay, returns recorded
> value
> > > > > without
> > > > > > re-firing fn. Defensible on crash recovery alone.
> > > > > >
> > > > > > Worked example (before/after code, mid-action case discussion):
> > > > > > https://gist.github.com/
> > > > > >
> > > > > > Open questions:
> > > > > >
> > > > > > 1. halt_when(predicate) as first-class transition primitive,
> or
> > > keep
> > > > > as
> > > > > > user-side conditional edge expression?
> > > > > > 2. __context.durable(key, fn) the right surface, or decorator?
> > > > > > 3. Anyone hitting the mid-action sequential case in
> production?
> > > > > >
> > > > > > Not proposing the suspend()-style mid-action pause in this
> thread.
> > > Want
> > > > > to
> > > > > > ship A and B, collect signal, revisit on a separate [DISCUSS] if
> > > demand
> > > > > > shows up.
> > > > > >
> > > > > > No deadline. Will follow up with [PROPOSAL] per piece once we
> > > converge.
> > > > > >
> > > > > > [1] https://github.com/apache/burr/pull/786
> > > > > >
> > > > > > Thanks, André
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
