One concern I have about Step 3 is the testing fidelity.

If users can install only the Task SDK and run a DAG locally, what level of
confidence can they get that the DAG will behave the same way in a real
Airflow deployment?

For example, how do we ensure that the DB-less runner and the production
Scheduler share the same execution semantics, especially for trigger rules,
task state transitions, and dynamic task mapping?

I also think we should consider the debugging experience when a DAG passes
SDK testing but fails in a real Airflow environment. Users should be able
to identify whether the failure comes from their task logic, the SDK
runtime, or Airflow-specific components.

Otherwise, users may still need to install and run a full Airflow
environment to gain confidence before deployment, which would significantly
reduce the value of Step 3.

How do you envision addressing this gap?


On 2026/11/16 04:39:18 Andrew Chang wrote:
> Hi Kaxil,
>
> Thanks for the detailed questions! They helped me to think about the scope
> more clearly. Here I want to clarify each of them:
>
> 1. What's the concrete failure?
> I reproduced some cases (Claude helped me investigate this). In each of
> them, when users follow the docs to write the test, the test passes, but
> the real run fails. Here are the detailed examples:
> https://gist.github.com/Andrushika/4d15594533993c05b1c3ef8d21350ae6
> So for the problems you listed, the first and the second kind are what
> users hit when they follow the docs. If they want a real run to avoid
this,
> they hit the third kind, because the only public way today is
`dag.test()`,
> and it needs a DB. So I think the three kinds are connected, and they can
> be addressed by the same approach that I am planning to propose.
>
> 2. Who is the user?
> Mainly Dag authors who test their own task logic, with or without agents.
> The needs of provider authors look more like our tests_common, so I would
> not focus on them here.
>
> 3. What is the unit under test?
> I am planning to split this into 3 steps. I will describe them below.
>
> 4. Which private interfaces are users relying on?
> TBH `ti.run()` is also the only one I know, plus one third-party plugin
> that follows our private code. I may have overstated this point in my
first
> mail.
>
> 5. Why isn't `dag.test()` enough?
> Most of its problems can be fixed, like the DB setup, the real retry
delay,
> and not giving the error back. So improving `dag.test()` is an option for
a
> whole Dag run. But two things are harder to fix. It always runs the
> upstream tasks, so users cannot test only one task. And it needs
> airflow-core and a DB, which #61803 (an issue opened by Amogh) says should
> go away for the client-server separation.
>
> 6. What's out of scope?
> Agreed. Calls to external systems are still the user's job to mock. I also
> don't plan to touch anything across Dags, or executors.
>
> What is in my mind is that we can split this into three steps:
> 1. Let users run a single task for real, completely without a DB. This is
> the smallest step, and it already solves most of the cases in my examples.
> 2. Make `dag.test()` better for writing tests. Today it requires a
prepared
> DB, waits for the real retry delay, and does not give the error back.
> 3. Let users run a whole Dag without core and without a DB. I think we
need
> this anyway if Dag authors only install the SDK one day. This is the
> biggest one, and I am not sure about the right design yet.
>
> I would also like to clarify one thing. I used the word "package" in the
> title and in my first mail, which could be misleading. Sorry for the
> confusion.
> What is in my mind is that step 1 and step 3 could be a separate module in
> the Task SDK, as a DB-less testing module. The design for step 3 might
need
> further discussion (e.g. how do we simulate the task state transitions? Do
> we need to share that logic between the scheduler and the SDK?). That's
why
> I think it might need an AIP.
>
> Hope this makes the direction clear. Looking forward to your thoughts!
>
> Thanks,
> Andrew
>
> Kaxil Naik <[email protected]> 於 2026年11月16日週一 上午7:41寫道:
>
> > Hi Andrew,
> >
> > Thanks for starting this. I think there's a real gap here, but before we
> > get to an AIP I'd like to pin down the exact problem and the scope,
because
> > "a testing package" can mean very different things.
> >
> > 1. What's the concrete failure? Can you share one or two real cases: a
Dag,
> > the test the user wrote, and what broke? There are a few different
problems
> > hiding in your list and they have different fixes:
> >
> >    - A test passes but the real run fails, because the user built the
> > context by hand. For example, a test that renders `{{ ds }}` with
> > `context={"ds": ...}` passes, but on Airflow 3 a run without a logical
date
> > has no `ds` in the context, so the template fails to render.
> >    - A test can't catch something only the task runner does, for
example a
> > TaskFlow return value that Airflow can't serialize into XCom.
> >    - A test can't be written at all without a metadata DB.
> >
> >    Which of these is the one you keep hitting?
> >
> > 2. Who is the user? Dag authors testing their own task logic, with or
> > without agents, or people writing custom operators and providers?
Jarek's
> > earlier thread was mostly about the second group, and their needs look a
> > lot like our tests_common.
> >
> > 3. What is the unit under test: one task, a few tasks with XCom between
> > them, or a whole Dag run (trigger rules, branching, mapped tasks)? Each
of
> > these is a very different amount of work.
> >
> > 4. Which private interfaces are users relying on today, and which ones
> > broke on upgrade? `ti.run()` (removed in 3.2) is the one I know of. Are
> > there others, like copies of `create_runtime_ti`?
> >
> > 5. Why isn't `dag.test()` enough? Is it that it needs a metadata DB,
that
> > it's slow, or that it gives you nothing to assert on at the task level?
If
> > it's only the DB or the speed, is improving `dag.test()` an option?
> >
> > 6. What's out of scope? I'd assume calls to external systems (S3, a
> > warehouse) are still the user's job to mock, even in your proposal.
> >
> > Answers to these, with real examples, would make it much easier to judge
> > whether this should be a new API, a change to `dag.test()`, or better
docs.
> >
> > Thanks,
> > Kaxil
> >
> > On Sun, 15 Nov 2026 at 19:45, Andrew Chang <[email protected]> wrote:
> >
> > > Hi all,
> > >
> > > I have been thinking about proposing a public testing package for
Airflow
> > > users. Before I spend more time on it, I would like to hear your
> > opinions.
> > >
> > > The problem I see is that there is no good way to test a Dag with
pytest
> > > today. The docs [1] suggest calling the operator directly with a
context
> > > you build by yourself, or running the whole Dag in a way that was made
> > for
> > > debugging.
> > >
> > > TBH I don't think either one is good enough.
> > >
> > > Why I think we need this:
> > >
> > > - If users build the context and fake the inputs by hand, they are
> > testing
> > > their own setup, not Airflow. The test still passes even when
Airflow's
> > > behavior changes.
> > >
> > > - Users who want a real test have to use our private interfaces. They
> > only
> > > find out it is broken after they upgrade.
> > >
> > > - A real test needs a prepared metadata database today, which is slow
and
> > > hard to set up in CI. A test for a single task should not need one.
(This
> > > might be related to the recent discussion on a faster `import airflow`
> > > [2].)
> > >
> > > - Many Dags are written with AI agents now. I think a fast and real
test
> > is
> > > what users will need most. It gives agents a quick way to verify their
> > > work.
> > >
> > > I found an earlier discussion started by Jarek [3], where releasing
our
> > > “whole internal test utilities” to users was rejected because of the
> > > compatibility burden. What I want to discuss here is different: a
public
> > > API designed for users, not our internal tools. And we already
maintain
> > the
> > > hard part, because our own tests run tasks this way.
> > >
> > > I would like to hear your thoughts. If it sounds reasonable, I would
be
> > > happy to write an AIP for it!
> > >
> > > Thanks,
> > > Andrew
> > >
> > > [1]
> > >
> > >
> >
https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#unit-tests
> > > [2] https://lists.apache.org/thread/1nhdc8y1x7dclcpcck98q5chzh1tvodk
> > > [3] https://lists.apache.org/thread/jonodvjnycv01qcsnlqczpvrjhfsx04g
> > >
> >
>

Reply via email to