Hi all,
I left a review on #786 with repros:
https://github.com/apache/burr/pull/786#pullrequestreview-5403771409.
One question fits better here, since it decides the scope vaquar asked
about.
Within an app and partition, the journal is selected by sequence_id, and
that moves. A resumed action runs at N+1, and so does an action retried
from a saved failed row. Loading the journal for the about-to-run
sequence addresses the crash-before-checkpoint case only. Covering
retries and repeated suspends needs an invocation identity that stays
fixed across retries and resumes, changes after a successful completion,
and stays distinct across partitions and child applications.
Is retry recovery in scope for this PR, and if so, what should that
identity be?
Sangkyoon Nam
On 2026-06-22 02:24 UTC vaquar khan wrote:
> Hi all,
>
> Thanks for looping me in, André, and thanks for putting this
together. It's
> a really nice piece of work. The suspend/resume flow is clean and the
> DeterminismError guard is a thoughtful touch. I left some detailed
notes on
> the PR, but since the design conversation is here, let me share one
> observation from testing plus where I land on the open questions.
>
> One thing from testing the branch that might be worth a look (crash
> recovery).
>
> The journal currently gets reloaded inside resume(), but not on the
normal
> recovery path (initialize_from(..., resume_at_next_action=True)). So if a
> process dies mid-action without a suspension, the memoized sub-steps seem
> to re-run on restart rather than replay. I put together a quick repro
to be
> sure I wasn't misreading it: a two-action graph, crash the second action
> after its first durable() call had journaled, then recover in a fresh
> process:
>
> loaded_journal on recovery: []
> manual load_journal(pk, app, seq): [('step_a', 'A')] # the row is there,
> key matches
> step_a re-fired on recovery? True
>
> The journal row is written and the key lines up (I double-checked the
> sequence_id), it just isn't loaded outside resume(). The PR's
exactly-once
> test passes because it goes through resume(), which is the path that
loads
> it. I mention it only because it touches the case 2 (crash recovery)
> framing. If the intended scope is deliberate suspend/resume, this is
> totally fine and just a docs note; if we want crashes covered too,
loading
> the journal on the recovery path looks like a small change. Curious what
> you all see as the intended scope.
>
> On André's Q1 (halt_when as a primitive vs conditional edges): I lean
> toward Stefan's instinct here. Keeping it expressed as state plus
> conditional edges feels like less new surface area, and
> halt_when(durable=[...]) as thin sugar over that, the way Elijah
suggested,
> seems like a nice middle ground that doesn't introduce a brand-new
> transition concept.
>
> On Q2 (the durable() API surface): both forms read well to me. The
callable
> form that's in the PR is the easiest to onboard and to unit-test,
which is
> worth something. Elijah's Azure-style yield ctx.durable(...) with
match is
> elegant and I can see the appeal, my only small hesitation is that
turning
> actions into generators changes the mental model a bit. If it's easy,
> supporting the callable form first and revisiting the generator form once
> people are using it might keep the initial surface small. Just a
> preference, not a strong view.
>
> On Elijah's "can we just model it as sub-state" question: I think that
> direction is reasonable, and the in-state path mostly already does
it. The
> one thing I noticed in testing is that the in-state journal isn't cleared
> after an action completes, so it accumulates across the run and the blob
> gets rewritten each step. That seems fixable within the in-state model
> (clear it on completion, keep only what the suspended action needs),
which
> to me actually supports Elijah's "keep it in state" instinct rather than
> arguing for a separate tier.
>
> On the two-tier / resume-once point: I'd also lean toward not solving
> resume concurrency in this PR, since Burr doesn't have a resume
concurrency
> model elsewhere. One small factual note for the record: the atomic
> primitive is already present per-persister (mark_suspension_resolved is a
> conditional update returning a bool), but the resume path reads
> record.resolved before the run and marks it after, so it isn't gating on
> that result yet. Happy to leave that out of scope for now and just
note in
> the docs that the current path is effectively single-writer.
>
> On hierarchy: just confirming what you already flagged, André, the
journal
> key is (partition_key, app_id, sequence_id, step_key) today with no
parent
> app id, so your plan to fold the parent app id in matches what's there.
>
> On Q3 (anyone hitting the mid-action sequential case in production): I'm
> not running this exact pattern in production myself, but the underlying
> cost (re-paying for an LLM batch on retry) is very familiar from data/ML
> pipeline work, so the motivation resonates.
>
> Stefan, on the gist idea, I'd be glad to help here . I can write up the
> loyalty-CRM batch-with-approval flow against the current branch and
include
> the crash repro above, so we have something concrete to talk UX around.
> Happy to start it if it's useful.
>
> Thanks again for driving this, it's a genuinely valuable feature and I'm
> glad to help however works best.
>
>
> Regards,
> Viquar Khan
> https://www.linkedin.com/in/vaquar-khan-b695577/
>
>
> On Sun, Jun 14, 2026 at 10:35 PM Elijah ben Izzy <
> [email protected]> wrote:
>
> > Not sure I agree with claude's points here will think it over:
> > 1. *State blob growth.* -- This depends on assumptions about
modeling it
> > that claude made -- I think that if we model it closer to state
it'll be
> > the same.
> > 2. *Resume-once under concurrency.* This is a whole can of worms we
> > don't really support a resume concurrency model anywhere? Why would
this be
> > special? We'd need to get a lock etc...
> > 3. *Queryability.* Sure could be a boolean flag that is by default
false
> > 4. *Schema coupling.* The state could be "waiting" effectively -- it
> > *is* state...
> >
> > not saying I disagree with the approach necessarily but worth thinking
> > through
> >
> > On Sun, Jun 7, 2026 at 3:10 PM Stefan Krawczyk <[email protected]>
> > wrote:
> >
> > > Can we start some gists? Or draft PRs? That would help me if we
talked UX
> > > and what it would mean. E.g. let's take that loyalty CRM case and
look at
> > > the API one would use to implement it.
> > >
> > > On Mon, Jun 1, 2026, 12:41 PM André Ahlert <[email protected]>
wrote:
> > >
> > > > Thanks Elijah, this is the right question to push on. Let me
answer it
> > > head
> > > > on, because the answer is "yes there is an easier way, and I
think we
> > > > should ship it as the default."
> > > >
> > > > You are right that a durable state can ride inside the existing
state
> > > model
> > > > rather than a parallel structure. PR #786 already proves this:
custom
> > > > persisters fall back to an in-state journal, and that path is
correct.
> > No
> > > > new persister methods, no dedicated tables, onboarding is zero.
For a
> > > > single-worker workflow it is the simplest thing that works, and
I do
> > not
> > > > want to force anyone onto a heavier model to get suspend/resume.
> > > >
> > > > So I want to reframe the design as two tiers rather than one
monolith:
> > > >
> > > > - *Default (light):* journal and suspension records live in the
> > > existing
> > > > state keyspace, exactly as you suggested. Any persister gets
durable
> > > > execution for free. This is the onboarding path.
> > > > - *Opt-in (heavy):* a persister that implements the dedicated
> > methods
> > > (
> > > > save_suspension, mark_suspension_resolved, etc.) for workloads that
> > > > outgrow the light path.
> > > >
> > > > The reason the opt-in tier earns its keep is not aesthetics, it
is four
> > > > concrete costs the in-state path hits at scale:
> > > >
> > > > 1. *State blob growth.* Journal entries and suspension payloads
> > inside
> > > > state mean every save reserializes the whole blob. A workflow
with N
> > > > memoized sub-steps rewrites O(N) state on every step. Fine for
small
> > > > flows,
> > > > painful for long ones.
> > > > 2. *Resume-once under concurrency.* Two workers resuming the same
> > > > suspension need an atomic claim. The SQL persisters expose exactly
> > > that
> > > > primitive: mark_suspension_resolved is a conditional UPDATE ...
> > WHERE
> > > > resolved = 0 returning rowcount > 0, so under concurrency exactly
> > one
> > > > caller wins. In-state cannot express that primitive at all. To be
> > > > precise
> > > > about the current state of the branch: resume() / aresume() still
> > > gate
> > > > on a non-atomic record.resolved read taken before the run and
> > discard
> > > > the bool, so the race is not yet closed end to end. Wiring
> > resumption
> > > to
> > > > skip the run unless mark_suspension_resolved returns True is
part of
> > > > this work, and it is only expressible on the dedicated tier (the
> > > > in-memory
> > > > persister is a plain loop, not atomic).
> > > > 3. *Queryability.* "List all pending suspensions awaiting approval"
> > is
> > > > one SQL query against a table. In-state it is a full scan of every
> > > app's
> > > > serialized state. Not viable for an ops dashboard or the Burr UI.
> > > > 4. *Schema coupling.* Mixing runtime metadata into the user state
> > > > keyspace invites key collisions and muddies "what is my state"
> > versus
> > > > "what
> > > > is runtime bookkeeping."
> > > >
> > > > *The real case driving this.* A loyalty CRM I am running in
> > production. A
> > > > campaign action generates a personalized offer per customer via
an LLM,
> > > > processed as a sequential batch of ~50. Two things break today:
> > > >
> > > > - Item 23 needs human approval before the offer goes out. Items 1
> > > > through 22 have already paid for their LLM calls. With halt_before/
> > > > halt_after I cannot pause mid batch, so I either fork state
manually
> > > or
> > > > re-run the batch and pay for 1 through 22 again.
> > > > - On any crash or retry mid batch, every offer regenerates. The
> > > > offer-generation LLM is the dominant cost in the workflow, so a
> > single
> > > > retry doubles the spend for that run.
> > > >
> > > > With __context.durable("offer", generate_offer, customer) the
replay
> > > reads
> > > > the recorded offers and re-fires nothing. This is test-backed
in the
> > PR:
> > > > the integration suite asserts a durable side effect runs
exactly once
> > > > across a suspend/resume cycle (the recorded value is replayed, the
> > > function
> > > > is not re-invoked). I also ran the HITL example end to end
against a
> > > local
> > > > model to confirm the same behavior outside the test harness.
> > > >
> > > > Now, here is where the in-state default would actually hurt
this case,
> > > > which is why I want the opt-in tier available: the recorded offer
> > > payloads
> > > > sit in the state blob and get reserialized on every subsequent
step,
> > and
> > > > the approval step is exactly where a second worker could pick
up the
> > same
> > > > suspension. Costs 1 and 2 are not hypothetical for this workload.
> > > >
> > > > *On the API surface.* I like your Azure-style yield
ctx.durable(...)
> > with
> > > > match. It reads cleanly. The current PR uses the callable form
> > > > ctx.durable(key,
> > > > fn, ...); I am open to the generator form if folks prefer it, the
> > journal
> > > > semantics are the same underneath. And halt_when(durable=[...])
as thin
> > > > sugar over conditional edges is a nice touch, that also answers
> > Stefan's
> > > > preference to keep halting expressed as state plus edges rather
than a
> > > new
> > > > transition primitive.
> > > >
> > > > *On Stefan's hierarchy point:* the journal key is (partition_key,
> > app_id,
> > > > sequence_id, step_key) today, with entries ordered by
call_index. It
> > does
> > > > not yet fold in a parent app id, so nested apps do not inherit
a parent
> > > > journal. I will make the key compose with the parent app id so
nested
> > > apps
> > > > inherit the journal correctly, which is the hierarchical reuse you
> > > flagged.
> > > >
> > > > Proposed path: keep suspend/resume and the journal in #786,
make the
> > > > in-state path the documented default, mark the dedicated persister
> > > methods
> > > > as the opt-in scale tier, and address hierarchy in the same PR.
If that
> > > > shape works for both of you I will split out anything that does not
> > need
> > > to
> > > > land together.
> > > >
> > > > No deadline. Will send a [PROPOSAL] once we converge.
> > > >
> > > > PS: sorry for the blogpost in dev list, but is a big theme. :)
> > > >
> > > > André
> > > >
> > > > Em dom., 31 de mai. de 2026 às 22:58, Elijah ben Izzy <
> > > > [email protected]> escreveu:
> > > >
> > > > > This is very cool. So I thought through there's definitely a need
> > here
> > > to
> > > > > pause mid-action.
> > > > >
> > > > > Some API ideas:
> > > > > 1. Put durable_keys as part of the declaration to match the
> > type-safe,
> > > > > declarative nature of the rest of it
> > > > > 2. Use something clever (i.e. match statements) to make it
easier to
> > > > match
> > > > > the mental model
> > > > >
> > > > > Azure durable functions has an interesting approach:
> > > > >
> > > > > def my_action(state, ctx):
> > > > > user = yield ctx.durable("fetch", db.get_user, state["uid"])
> > > > > match user.tier:
> > > > > case "gold":
> > > > > result = yield ctx.durable("gold_flow", gold_flow,
> > user)
> > > > > case _:
> > > > > result = yield ctx.durable("std_flow", std_flow, user)
> > > > > return state.update(result=result)
> > > > >
> > > > > Worth exploring. I'm also wondering if there's a simple way
to add an
> > > > > optional field to the data model rather than a whole new
"supports
> > > > > durable"? I.E. would there be an easier way to onboard? Could we
> > model
> > > it
> > > > > as some sort of sub-state that we could store and pass in? If
we had
> > > > > durable things as a special key-space in the state could we
just use
> > > the
> > > > > same ones? Trade-offs not sure the right one.
> > > > >
> > > > > The problem is that if we don't have suspend() we decouple it but
> > need
> > > a
> > > > > mechanism to halt execution. Might be a good place for syntactic
> > sugar
> > > --
> > > > > halt_during? halt_when(durable=["key"])?
> > > > >
> > > > > With iterator-based APIs it's actually up to the user to
handle the
> > > > halting
> > > > > -- this completely bypasses this issue (we just send out a
> > durable-set
> > > > > event mid-stride as we would an iterator or an iterator of
streams).
> > > > >
> > > > >
> > > > >
> > > > > On Sun, May 31, 2026 at 11:45 AM Stefan Krawczyk <
> > > > > [email protected]>
> > > > > wrote:
> > > > >
> > > > > > If halt_when has clear semantics if what happens next then that
> > > sounds
> > > > > > good. But I haven't found a clear way to make that obvious.
So I
> > > think
> > > > > > making users set up state and conditional edges should be the
> > cleaner
> > > > > model
> > > > > > (we should also show that running a graph with halt* is fine in
> > > > examples
> > > > > > and drop that warning).
> > > > > >
> > > > > > Durable key seems fine I think. I think the hierarchical nature
> > > should
> > > > be
> > > > > > utilized that we've set up, so more capabilities to enable
that way
> > > of
> > > > > > pausing and resuming to be more useful makes sense to me --
IIUC.
> > > > > >
> > > > > > On Thu, May 28, 2026, 6:14 AM André Ahlert <[email protected]>
> > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > PR #786 <https://github.com/apache/burr/pull/786> [1]
proposes
> > two
> > > > > > > additions to handle long-running agent workloads. Moving the
> > design
> > > > > > > question from DM with Stefan onto the list before we iterate
> > > further.
> > > > > > >
> > > > > > > Gap today: halt_before / halt_after only stop between
actions.
> > Two
> > > > > cases
> > > > > > > keep showing up:
> > > > > > >
> > > > > > > 1. HITL inside one action (e.g. sequential batch where
item 23
> > > of
> > > > 50
> > > > > > > needs approval, items 1-22 already paid for LLM calls).
> > > > > > > 2. Crash recovery mid-action without re-firing paid
sub-steps.
> > > > > > >
> > > > > > > Langgraph covers both w
[message truncated...]
