Hello everyone,
This is the follow-up to the discussion I reopened on 2026-10-27 [1].
In that thread I proposed to stop treating changes to how we develop
and test Airflow (CI, development environment, test infrastructure)
as AIPs, and to give them a separate place on the wiki. Nobody
objected so far, but since no-objection does not mean consensus,
as we learned recently - before I start a formal vote, i wanted to turn it
into a PROPOSAL so that it has clear subject and separate thread.
Why
---
The AIP page [2], in "What is considered a major change that needs
an AIP?", gives two criteria. An AIP is needed for:
* "Any major new feature, subsystem, or piece of functionality"
* "Any change that impacts the public interfaces of the project"
and it lists what the public interfaces are:
* Airflow DAG DSL components: DAG, XCom, TaskInstance and context
* The Web UI
* Core operators (BaseOperator, SubDagOperator, SensorOperator)
* REST endpoints
* Command line tools and arguments
None of the proposals listed below meets either criterion:
* They are not a feature, subsystem or functionality of Airflow.
Nothing they introduced ships to users in any Airflow release:
the linter setup, Breeze, the CI image, the move away from
Travis, pytest, the system and performance test frameworks, the
self-hosted runners, the guardrails of the AWS account that hosts
them, and running CI in contributors' forks all live in our
repository and infrastructure, not in the product.
* They change none of the interfaces listed above. No Dag authoring
component, UI page, operator, REST endpoint or CLI command
behaves differently because of any of them.
What they do change is how we, contributors and maintainers, work:
how code is checked, built and tested, and where CI runs. Those are
decisions about how the project operates. We made them AIPs only
because the AIP page was the one place we had to describe a design
and record a decision. As the AIP-120 discussion showed, that
causes confusion: about what is being decided, about who is
affected, and about which kind of vote applies. An AIP vote on a
code change lets a single binding -1 block it, while a decision
about how the project operates is a procedural, majority vote [3].
Moving them to their own section makes the AIP list again what the
AIP page says it is: changes to Airflow itself and its public
interfaces.
What I propose
--------------
1. A new wiki section, next to the AIPs: "ADIP - Airflow Development
Improvement Proposals". The name is a placeholder; if you have a
better one, say so in this thread and we will go with the one most
people prefer.
2. ADIPs cover proposals that only change how the project is
developed, tested and built: CI, the development environment,
static checks, test infrastructure and the infrastructure that
runs CI. Anything that affects users (installation, images,
providers, public interfaces) stays an AIP.
3. ADIPs are decided like any other decision about how the project
operates: discussion on dev@, then [LAZY CONSENSUS] or a
procedural [VOTE] (majority of binding votes, at least 3 binding
+1 and more binding +1 than -1) [3].
4. The CI / development AIPs listed below move to the new section and
are renumbered from 1. Nothing about any decision already made
changes: each page keeps its content, history and discussion
links.
5. Old links keep working:
- every moved page keeps its Confluence page ID, so all
".../pages/<id>/..." links (the form used in most of our mails)
resolve to the moved page;
- every new title says "(formerly AIP-N)";
- a stub page stays under each old title, saying "Moved to
ADIP-N" and linking to the new page, so links by the old title
and old bookmarks still land somewhere useful;
- the old AIP numbers are retired and never reused.
6. The AIP index page gets a short note pointing to the ADIP section.
The index lists are generated from page labels, so the moved pages
get an "adip" label instead of the AIP state labels, and they drop
out of the AIP lists automatically.
Proposed renames
----------------
Numbered in the order of the original AIP numbers.
ADIP-1 (formerly AIP-4) Automation of System Tests
was: AIP-4 Automation of System Tests [Deps: AIP-47]
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/93324355
ADIP-2 (formerly AIP-6) Enforce the usage of a code formatter & linter
was: AIP-6 Enforce the usage of a code formatter & linter
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/99844429
ADIP-3 (formerly AIP-7) Simplified development workflow
was: AIP-7 Simplified development workflow
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/99844641
ADIP-4 (formerly AIP-9) Automated Dependency Management
was: AIP-9 Automated Dependency Management
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/100828074
ADIP-5 (formerly AIP-10) Multi-layered and multi-stage official Airflow
CI image
was: AIP-10 Multi-layered and multi-stage official Airflow CI image
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/100828709
ADIP-6 (formerly AIP-23) Migrate out of Travis CI
was: AIP-23 Migrate out of Travis CI
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/122916333
ADIP-7 (formerly AIP-27) Migrate to pytest
was: AIP-27 Migrate to pytest
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/132321242
ADIP-8 (formerly AIP-47) New design of Airflow System Tests
was: AIP-47 New design of Airflow System Tests
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/199532973
ADIP-9 (formerly AIP-59) Performance tests framework
was: AIP-59 Performance tests framework
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/278465469
ADIP-10 (formerly AIP-118) Self-hosted runners for Apache Airflow CI
was: AIP-118 Self-hosted runners for Apache Airflow CI
https://cwiki.apache.org/confluence/x/-JXwGg
(its appendix page moves with it, as
"ADIP-9 Appendix - evidence and implementation details")
ADIP-11 (formerly AIP-119) Anomaly monitoring and guardrails for the
Airflow AWS account
was: AIP-119 Anomaly monitoring and guardrails for the Airflow AWS
account
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974695
ADIP-12 (formerly AIP-120) Run CI for external contributions in contributor
forks
was: AIP-120 Run CI for external contributions in contributor forks
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974711
These stay AIPs, although they are close to the line, because they
affect users: AIP-8 (providers as separate packages), AIP-11 (landing
page), AIP-26 (production Docker image) and AIP-95 (provider
lifecycle).
AIP-47 (System tests) is the most borderline of the moved items, since
User docs take their examples from system tests. If you think it,
or any other entry, belongs on the other side, please say so in the
thread and I will adjust the list.
The vote proposal
--------
This would be a procedural vote: majority of binding votes, at least 3
binding +1 and more binding +1 than -1 [3]. Everyone is welcome to
vote; votes from PMC members are binding.
I realise this is a busy time: the 3.4.0 release vote, provider
releases, the Summit Reconnect on 4-5 November. If you need more time
to have a look, say so, and I would like to start a vote on it this week.
But not by
much: I fear we will have to act quickly in the "Your PR, your CI"
area (AIP-120, which would become ADIP-12), given the pressure on the
shared ASF runners and on our reviewers, and our quest to limit
big PRs and encourage small ones.
For the same reason, I would like to start the vote on "Your PR, your CI"
(formerly AIP-120 - possibly future ADIP-12) soon,
without waiting for this one to complete. The two are
independent: that vote decides whether we run CI for external
contributions in contributors' forks; this one only decides where
such proposals live on the wiki and how they are numbered.
Let me know if that proposal makes sense and whether you have
any concerns.
J.
[1] https://lists.apache.org/thread/h5qsfqthmsqoh5j7wyy12r90mnnvz8kt
[2]
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
[3] https://www.apache.org/foundation/voting.html
