Hi Kaxil,

Thanks for the detailed questions! They helped me to think about the scope
more clearly. Here I want to clarify each of them:

1. What's the concrete failure?
I reproduced some cases (Claude helped me investigate this). In each of
them, when users follow the docs to write the test, the test passes, but
the real run fails. Here are the detailed examples:
https://gist.github.com/Andrushika/4d15594533993c05b1c3ef8d21350ae6
So for the problems you listed, the first and the second kind are what
users hit when they follow the docs. If they want a real run to avoid this,
they hit the third kind, because the only public way today is `dag.test()`,
and it needs a DB. So I think the three kinds are connected, and they can
be addressed by the same approach that I am planning to propose.

2. Who is the user?
Mainly Dag authors who test their own task logic, with or without agents.
The needs of provider authors look more like our tests_common, so I would
not focus on them here.

3. What is the unit under test?
I am planning to split this into 3 steps. I will describe them below.

4. Which private interfaces are users relying on?
TBH `ti.run()` is also the only one I know, plus one third-party plugin
that follows our private code. I may have overstated this point in my first
mail.

5. Why isn't `dag.test()` enough?
Most of its problems can be fixed, like the DB setup, the real retry delay,
and not giving the error back. So improving `dag.test()` is an option for a
whole Dag run. But two things are harder to fix. It always runs the
upstream tasks, so users cannot test only one task. And it needs
airflow-core and a DB, which #61803 (an issue opened by Amogh) says should
go away for the client-server separation.

6. What's out of scope?
Agreed. Calls to external systems are still the user's job to mock. I also
don't plan to touch anything across Dags, or executors.

What is in my mind is that we can split this into three steps:
1. Let users run a single task for real, completely without a DB. This is
the smallest step, and it already solves most of the cases in my examples.
2. Make `dag.test()` better for writing tests. Today it requires a prepared
DB, waits for the real retry delay, and does not give the error back.
3. Let users run a whole Dag without core and without a DB. I think we need
this anyway if Dag authors only install the SDK one day. This is the
biggest one, and I am not sure about the right design yet.

I would also like to clarify one thing. I used the word "package" in the
title and in my first mail, which could be misleading. Sorry for the
confusion.
What is in my mind is that step 1 and step 3 could be a separate module in
the Task SDK, as a DB-less testing module. The design for step 3 might need
further discussion (e.g. how do we simulate the task state transitions? Do
we need to share that logic between the scheduler and the SDK?). That's why
I think it might need an AIP.

Hope this makes the direction clear. Looking forward to your thoughts!

Thanks,
Andrew

Kaxil Naik <[email protected]> 於 2026年11月16日週一 上午7:41寫道:

> Hi Andrew,
>
> Thanks for starting this. I think there's a real gap here, but before we
> get to an AIP I'd like to pin down the exact problem and the scope, because
> "a testing package" can mean very different things.
>
> 1. What's the concrete failure? Can you share one or two real cases: a Dag,
> the test the user wrote, and what broke? There are a few different problems
> hiding in your list and they have different fixes:
>
>    - A test passes but the real run fails, because the user built the
> context by hand. For example, a test that renders `{{ ds }}` with
> `context={"ds": ...}` passes, but on Airflow 3 a run without a logical date
> has no `ds` in the context, so the template fails to render.
>    - A test can't catch something only the task runner does, for example a
> TaskFlow return value that Airflow can't serialize into XCom.
>    - A test can't be written at all without a metadata DB.
>
>    Which of these is the one you keep hitting?
>
> 2. Who is the user? Dag authors testing their own task logic, with or
> without agents, or people writing custom operators and providers? Jarek's
> earlier thread was mostly about the second group, and their needs look a
> lot like our tests_common.
>
> 3. What is the unit under test: one task, a few tasks with XCom between
> them, or a whole Dag run (trigger rules, branching, mapped tasks)? Each of
> these is a very different amount of work.
>
> 4. Which private interfaces are users relying on today, and which ones
> broke on upgrade? `ti.run()` (removed in 3.2) is the one I know of. Are
> there others, like copies of `create_runtime_ti`?
>
> 5. Why isn't `dag.test()` enough? Is it that it needs a metadata DB, that
> it's slow, or that it gives you nothing to assert on at the task level? If
> it's only the DB or the speed, is improving `dag.test()` an option?
>
> 6. What's out of scope? I'd assume calls to external systems (S3, a
> warehouse) are still the user's job to mock, even in your proposal.
>
> Answers to these, with real examples, would make it much easier to judge
> whether this should be a new API, a change to `dag.test()`, or better docs.
>
> Thanks,
> Kaxil
>
> On Sun, 15 Nov 2026 at 19:45, Andrew Chang <[email protected]> wrote:
>
> > Hi all,
> >
> > I have been thinking about proposing a public testing package for Airflow
> > users. Before I spend more time on it, I would like to hear your
> opinions.
> >
> > The problem I see is that there is no good way to test a Dag with pytest
> > today. The docs [1] suggest calling the operator directly with a context
> > you build by yourself, or running the whole Dag in a way that was made
> for
> > debugging.
> >
> > TBH I don't think either one is good enough.
> >
> > Why I think we need this:
> >
> > - If users build the context and fake the inputs by hand, they are
> testing
> > their own setup, not Airflow. The test still passes even when Airflow's
> > behavior changes.
> >
> > - Users who want a real test have to use our private interfaces. They
> only
> > find out it is broken after they upgrade.
> >
> > - A real test needs a prepared metadata database today, which is slow and
> > hard to set up in CI. A test for a single task should not need one. (This
> > might be related to the recent discussion on a faster `import airflow`
> > [2].)
> >
> > - Many Dags are written with AI agents now. I think a fast and real test
> is
> > what users will need most. It gives agents a quick way to verify their
> > work.
> >
> > I found an earlier discussion started by Jarek [3], where releasing our
> > “whole internal test utilities” to users was rejected because of the
> > compatibility burden. What I want to discuss here is different: a public
> > API designed for users, not our internal tools. And we already maintain
> the
> > hard part, because our own tests run tasks this way.
> >
> > I would like to hear your thoughts. If it sounds reasonable, I would be
> > happy to write an AIP for it!
> >
> > Thanks,
> > Andrew
> >
> > [1]
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#unit-tests
> > [2] https://lists.apache.org/thread/1nhdc8y1x7dclcpcck98q5chzh1tvodk
> > [3] https://lists.apache.org/thread/jonodvjnycv01qcsnlqczpvrjhfsx04g
> >
>

Reply via email to