Really good news! The timelines seemed very unrealistic when AF3 was announced during Airflow Summit 2024, but the community decked all hands together and helped get to this stage, and that too with really good quality levels.
Exciting times ahead for Airflow, and I am really proud to be part of this amazing community and collaborating on Airflow 3! Thanks & Regards, Amogh Desai On Sat, Feb 1, 2025 at 1:31 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Woooho! > > On Sat, Feb 1, 2025 at 2:21 AM Vikram Koka <vik...@astronomer.io.invalid> > wrote: > > > This is fantastic! Congratulations to everyone on this huge milestone. > > Thank you all for your hard work in getting us to this point. I am truly > > proud to be a part of this team. > > > > Best regards, > > Vikram > > > > > > On Fri, Jan 31, 2025 at 5:17 PM Jed Cunningham <jedcunning...@apache.org > > > > wrote: > > > > > Dear Airflow Community, > > > > > > I am thrilled to announce the availability of Apache Airflow > 3.0.0.alpha1 > > > for testing! Airflow 3.0 marks a significant milestone as the first > major > > > release in over four years, introducing improvements that enhance user > > > experience, task execution, and system scalability. > > > > > > First, a few caveats: > > > > > > This is an alpha release, so do not run it in production. It may > contain > > > significant issues, and you will likely need to reset your database > > between > > > this and subsequent alpha, beta or release candidate versions. > (Consider > > > yourself warned!) > > > > > > This release is intended for Airflow developers and early adopters to > > test > > > the build and start preparing for Airflow 3.0.0. This is not an > official > > > release—that will happen when we create a release candidate and hold a > > > vote. The expected timeline for the first release candidate is the week > > of > > > 2025-03-31, but we encourage early feedback to help stabilize the > > release. > > > > > > What's new in 3.0.0.alpha1? > > > > > > Airflow 3.0.0 introduces significant enhancements and breaking changes. > > > > > > Notable Features > > > > > > DAG versioning & Bundles > > > Airflow now tracks DAG versions, offering better visibility into > > historical > > > DAG changes and execution states. The introduction of DAG Bundles > ensures > > > tasks run with the correct code version, even as DAGs evolve. > > > > > > Modern Web Application > > > The UI has been rebuilt using React and a complete API-driven > structure, > > > improving maintainability and extensibility. It includes a new > > > component-based design system and an enhanced information > architecture. A > > > new React-based plugin system supports custom widgets, improved > workflow > > > visibility, and integration with external tools. > > > > > > Task Execution Interface > > > Airflow 3.0 adopts a client / server architecture, decoupling task > > > execution from the internal meta-database via API-based interaction. > This > > > allows for remote execution across networks, multi-language support, > > > enhanced security, and better dependency management. The Edge Executor > > > further enables seamless remote task execution without direct database > > > connections. > > > > > > Data Assets & Asset-Centric Syntax > > > Airflow 3.0 enhances dataset management by introducing Data Assets, > > > expanding beyond tables and files to include ML models and more. Assets > > can > > > be explicitly defined using the @asset decorator, simplifying tracking > > and > > > dependencies. > > > > > > External Event-Driven Scheduling > > > > > > Airflow now supports event-driven DAG triggers from external sources > like > > > message queues. This builds upon dataset scheduling and enhances > > > integration with the external data ecosystem. > > > > > > For a more comprehensive list of new features, please see the > 3.0.0apha1 > > > release notes: > > > > > > > > > https://github.com/apache/airflow/blob/3.0.0a1/RELEASE_NOTES.rst#airflow-300alpha1-2025-01-31 > > > > > > For a list of the breaking changes, please see visit: > > > https://cwiki.apache.org/confluence/x/9pCMEw > > > > > > Known issues in 3.0.0.alpha1: > > > > > > - > > > > > > AIP-72 - Task Execution Interface > > > - > > > > > > DB isolation in tasks is not enforced yet > > > - > > > > > > KubernetesExecutor does not yet use the TaskExecution Interface > > > - > > > > > > The following does not work on LocalExecutor or CeleryExecutor: > > > Dynamic Task Mapping, Skip based tasks (e.g. branch or skip > > > operators), and > > > Task callbacks. > > > - > > > > > > OpenLineage listeners are not yet functional in this alpha > release. > > > - > > > > > > AIP-38 - Modern Web Application > > > - > > > > > > The new UI has limited functionality at this time and is still > > being > > > enhanced until GA. However, feedback on the UX flow is > > > appreciated at this > > > time. > > > - > > > > > > The underlying FastAPI API server, including the new UI and > public > > > API, does not yet have auth. > > > - > > > > > > UI may be slow if you have a lot of DAGs or very big DAGs > > > - > > > > > > Notable areas of functionality that are not yet functional in the > > UI: > > > DAG Versioning, Data Assets, Backfills, and the DAG Gantt View. > > > - > > > > > > AIP-63 - DAG Versioning > > > - > > > > > > Callbacks are not yet working on the DAG Processor (e.g. DAG > level > > > callbacks) > > > - > > > > > > The concept of a separate `--subdir` in the DAG processor is > being > > > replaced with separate DAG bundles, but currently the DAG > > > processor always > > > parses all DAG bundles. > > > - > > > > > > AIP-82 - Event Based Scheduling > > > - > > > > > > Scheduling using some triggers, for example file-based, can lead > to > > > duplicate DAG runs. > > > > > > > > > Where to get it? > > > > > > The alpha snapshot is available at: > > > https://dist.apache.org/repos/dist/dev/airflow/3.0.0a1/ > > > > > > - > > > > > > apache-airflow-3.0.0a1-bin.tar.gz: Binary Python "sdist" snapshot. > > > - > > > > > > apache_airflow-3.0.0a1-py3-none-any.whl: Binary Python wheel > snapshot. > > > > > > > > > This snapshot has not been published to PyPI. > > > > > > Also present are alpha releases for apache-airflow-task-sdk, > > > apache-airflow-providers-standard, apache-airflow-providers-fab, and > > > apache-airflow-providers-celery, which you will also need for the > 3.0.0a1 > > > to work. These have also not been published to PyPI. > > > > > > Public Keys & Verification > > > > > > Public keys for verification are available at: > > > https://www.apache.org/dist/airflow/KEYS > > > > > > Instructions: > > > > > > You can build a virtualenv that installs this alpha, and other required > > > packages (e.g. task sdk), like this: > > > > > > ``` > > > > > > uv venv > > > > > > uv pip install --find-links > > > https://dist.apache.org/repos/dist/dev/airflow/3.0.0a1/ > > > apache-airflow==3.0.0a1 apache-airflow-task-sdk==1.0.0a1 > > > apache-airflow-providers-standard==0.1.0a1 > > > apache-airflow-providers-fab==2.0.0a1 > > > apache-airflow-providers-celery==3.11.0a1 > > > > > > ``` > > > > > > Below are some of the changes you’ll need to consider in order to run > > this > > > Alpha release: > > > > > > - > > > > > > The standalone DAG processor is now required, and can be started > with > > > `airflow dag-processor`. > > > - > > > > > > The new UI and public API is started by running `airflow > fastapi-api`, > > > and the UI is available on port `9091` and path `/webapp`. > > > - > > > > > > Depending on your deployment setup, you may need to set the > `[workers] > > > execution_api_server_url` config option. This defaults to ` > > > http://localhost:9091/execution/` > <http://localhost:9091/execution/> <http://localhost:9091/execution/> > > <http://localhost:9091/execution/>. > > > > > > > > > Get Involved > > > > > > We encourage the community to test this release and report any issues > or > > > feedback. Your contributions help us ensure a stable and reliable > Airflow > > > 3.0.0 release. > > > > > > A huge thank you to all the contributors who have worked on this > > milestone > > > release! > > > > > > Best, > > > Utkarsh and Jed > > > > > >
