Lee-W commented on code in PR #62440:
URL: https://github.com/apache/airflow/pull/62440#discussion_r2850403550
##########
AGENTS.md:
##########
@@ -3,29 +3,100 @@
# AGENTS instructions
-The main developer documentation lives in the `contributing-docs` directory.
The following points summarise
-how to set up the environment, run checks, build docs and follow the PR
workflow.
+## Environment Setup
-## Local virtualenv and Breeze
+- Install prek: `uv tool install prek`
+- Enable commit hooks: `prek install`
+- **Never run pytest, python, or airflow commands directly on the host** —
always use `breeze`.
+- Place temporary scripts in `dev/` (mounted as `/opt/airflow/dev/` inside
Breeze).
-- [`07_local_virtualenv.rst`](contributing-docs/07_local_virtualenv.rst)
explains how to prepare a local Python environment using `uv`. The tool creates
and syncs a `.venv` and installs dependencies with commands such as `uv venv`
and `uv sync`.
--
[`06_development_environments.rst`](contributing-docs/06_development_environments.rst)
compares the local virtualenv with the Docker based Breeze environment. Breeze
replicates CI and includes services like databases for integration tests.
+## Commands
-## Prek hooks
+- **Run a single test:** `breeze run pytest
path/to/test.py::TestClass::test_method -xvs`
+- **Run a test file:** `breeze run pytest path/to/test.py -xvs`
+- **Run a Python script:** `breeze run python dev/my_script.py`
+- **Run Airflow CLI:** `breeze run airflow dags list`
+- **Type-check:** `breeze run mypy path/to/code`
+- **Lint/format (runs on host):** `prek run --all-files`
+- **Lint with ruff only:** `prek run ruff --all-files`
+- **Format with ruff only:** `prek run ruff-format --all-files`
+- **Build docs:** `breeze build-docs`
-- Installation and usage of `prek` are described in
[`03a_contributors_quick_start_beginners.rst`](contributing-docs/03a_contributors_quick_start_beginners.rst).
Install with `uv tool install prek` and run checks via `prek --all-files`.
-- [`08_static_code_checks.rst`](contributing-docs/08_static_code_checks.rst)
provides more details on the available hooks and prerequisites. Enable the
hooks with `prek install` so they run automatically on each commit.
+SQLite is the default backend. Use `--backend postgres` or `--backend mysql`
for integration tests that need those databases. If Docker networking fails,
run `docker network prune`.
-## Running tests
+## Repository Structure
--
[`03a_contributors_quick_start_beginners.rst`](contributing-docs/03a_contributors_quick_start_beginners.rst)
shows running tests inside Breeze. Use `pytest` inside the container for
individual files or invoke `breeze testing` commands to run full suites, e.g.
`breeze --backend postgres --python 3.10 testing tests --test-type All`.
+UV workspace monorepo. Key paths:
-## Building documentation
+- `airflow-core/src/airflow/` — core scheduler, API, CLI, models
+ - `models/` — SQLAlchemy models (DAG, TaskInstance, DagRun, Asset, etc.)
+ - `jobs/` — scheduler, triggerer, DAG processor runners
+ - `api_fastapi/core_api/` — public REST API v2, UI endpoints
+ - `api_fastapi/execution_api/` — task execution communication API
+ - `dag_processing/` — DAG parsing and validation
+ - `cli/` — command-line interface
+ - `ui/` — React/TypeScript web interface (Vite)
+- `task-sdk/` — lightweight SDK for DAG authoring and task execution runtime
Review Comment:
```suggestion
- `task-sdk/` — lightweight SDK for Dag authoring and task execution runtime
```
##########
AGENTS.md:
##########
@@ -3,29 +3,100 @@
# AGENTS instructions
-The main developer documentation lives in the `contributing-docs` directory.
The following points summarise
-how to set up the environment, run checks, build docs and follow the PR
workflow.
+## Environment Setup
-## Local virtualenv and Breeze
+- Install prek: `uv tool install prek`
+- Enable commit hooks: `prek install`
+- **Never run pytest, python, or airflow commands directly on the host** —
always use `breeze`.
+- Place temporary scripts in `dev/` (mounted as `/opt/airflow/dev/` inside
Breeze).
-- [`07_local_virtualenv.rst`](contributing-docs/07_local_virtualenv.rst)
explains how to prepare a local Python environment using `uv`. The tool creates
and syncs a `.venv` and installs dependencies with commands such as `uv venv`
and `uv sync`.
--
[`06_development_environments.rst`](contributing-docs/06_development_environments.rst)
compares the local virtualenv with the Docker based Breeze environment. Breeze
replicates CI and includes services like databases for integration tests.
+## Commands
-## Prek hooks
+- **Run a single test:** `breeze run pytest
path/to/test.py::TestClass::test_method -xvs`
+- **Run a test file:** `breeze run pytest path/to/test.py -xvs`
+- **Run a Python script:** `breeze run python dev/my_script.py`
+- **Run Airflow CLI:** `breeze run airflow dags list`
+- **Type-check:** `breeze run mypy path/to/code`
+- **Lint/format (runs on host):** `prek run --all-files`
+- **Lint with ruff only:** `prek run ruff --all-files`
+- **Format with ruff only:** `prek run ruff-format --all-files`
+- **Build docs:** `breeze build-docs`
-- Installation and usage of `prek` are described in
[`03a_contributors_quick_start_beginners.rst`](contributing-docs/03a_contributors_quick_start_beginners.rst).
Install with `uv tool install prek` and run checks via `prek --all-files`.
-- [`08_static_code_checks.rst`](contributing-docs/08_static_code_checks.rst)
provides more details on the available hooks and prerequisites. Enable the
hooks with `prek install` so they run automatically on each commit.
+SQLite is the default backend. Use `--backend postgres` or `--backend mysql`
for integration tests that need those databases. If Docker networking fails,
run `docker network prune`.
-## Running tests
+## Repository Structure
--
[`03a_contributors_quick_start_beginners.rst`](contributing-docs/03a_contributors_quick_start_beginners.rst)
shows running tests inside Breeze. Use `pytest` inside the container for
individual files or invoke `breeze testing` commands to run full suites, e.g.
`breeze --backend postgres --python 3.10 testing tests --test-type All`.
+UV workspace monorepo. Key paths:
-## Building documentation
+- `airflow-core/src/airflow/` — core scheduler, API, CLI, models
+ - `models/` — SQLAlchemy models (DAG, TaskInstance, DagRun, Asset, etc.)
+ - `jobs/` — scheduler, triggerer, DAG processor runners
+ - `api_fastapi/core_api/` — public REST API v2, UI endpoints
+ - `api_fastapi/execution_api/` — task execution communication API
+ - `dag_processing/` — DAG parsing and validation
+ - `cli/` — command-line interface
+ - `ui/` — React/TypeScript web interface (Vite)
+- `task-sdk/` — lightweight SDK for DAG authoring and task execution runtime
+ - `src/airflow/sdk/execution_time/` — task runner, supervisor
+- `providers/` — 100+ provider packages, each with its own `pyproject.toml`
+- `airflow-ctl/` — management CLI tool
+- `chart/` — Helm chart for Kubernetes deployment
-- Documentation can be built locally using `uv run --group docs build-docs` as
described in
[`11_documentation_building.rst`](contributing-docs/11_documentation_building.rst).
Within Breeze the equivalent command is `breeze build-docs`.
+## Architecture Boundaries
-## Pull request guidelines
+1. Users author DAGs with the Task SDK (`airflow.sdk`).
+2. DAG Processor parses DAG files in isolated processes and stores serialized
DAGs in the metadata DB.
+3. Scheduler reads serialized DAGs — **never runs user code** — and creates
DAG runs / task instances.
+4. Workers execute tasks via Task SDK and communicate with the API server
through the Execution API — **never access the metadata DB directly**.
+5. API Server serves the React UI and handles all client-database interactions.
+6. Triggerer evaluates deferred tasks/sensors in isolated processes.
Review Comment:
```suggestion
1. Users author Dags with the Task SDK (`airflow.sdk`).
2. Dag Processor parses Dag files in isolated processes and stores
serialized DAGs in the metadata DB.
3. Scheduler reads serialized Dags — **never runs user code** — and creates
Dag runs / task instances.
4. Workers execute tasks via Task SDK and communicate with the API server
through the Execution API — **never access the metadata DB directly**.
5. API Server serves the React UI and handles all client-database
interactions.
6. Triggerer evaluates deferred tasks/sensors in isolated processes.
```
##########
AGENTS.md:
##########
@@ -3,29 +3,100 @@
# AGENTS instructions
-The main developer documentation lives in the `contributing-docs` directory.
The following points summarise
-how to set up the environment, run checks, build docs and follow the PR
workflow.
+## Environment Setup
-## Local virtualenv and Breeze
+- Install prek: `uv tool install prek`
+- Enable commit hooks: `prek install`
+- **Never run pytest, python, or airflow commands directly on the host** —
always use `breeze`.
+- Place temporary scripts in `dev/` (mounted as `/opt/airflow/dev/` inside
Breeze).
-- [`07_local_virtualenv.rst`](contributing-docs/07_local_virtualenv.rst)
explains how to prepare a local Python environment using `uv`. The tool creates
and syncs a `.venv` and installs dependencies with commands such as `uv venv`
and `uv sync`.
--
[`06_development_environments.rst`](contributing-docs/06_development_environments.rst)
compares the local virtualenv with the Docker based Breeze environment. Breeze
replicates CI and includes services like databases for integration tests.
+## Commands
-## Prek hooks
+- **Run a single test:** `breeze run pytest
path/to/test.py::TestClass::test_method -xvs`
+- **Run a test file:** `breeze run pytest path/to/test.py -xvs`
+- **Run a Python script:** `breeze run python dev/my_script.py`
+- **Run Airflow CLI:** `breeze run airflow dags list`
+- **Type-check:** `breeze run mypy path/to/code`
+- **Lint/format (runs on host):** `prek run --all-files`
+- **Lint with ruff only:** `prek run ruff --all-files`
+- **Format with ruff only:** `prek run ruff-format --all-files`
+- **Build docs:** `breeze build-docs`
-- Installation and usage of `prek` are described in
[`03a_contributors_quick_start_beginners.rst`](contributing-docs/03a_contributors_quick_start_beginners.rst).
Install with `uv tool install prek` and run checks via `prek --all-files`.
-- [`08_static_code_checks.rst`](contributing-docs/08_static_code_checks.rst)
provides more details on the available hooks and prerequisites. Enable the
hooks with `prek install` so they run automatically on each commit.
+SQLite is the default backend. Use `--backend postgres` or `--backend mysql`
for integration tests that need those databases. If Docker networking fails,
run `docker network prune`.
-## Running tests
+## Repository Structure
--
[`03a_contributors_quick_start_beginners.rst`](contributing-docs/03a_contributors_quick_start_beginners.rst)
shows running tests inside Breeze. Use `pytest` inside the container for
individual files or invoke `breeze testing` commands to run full suites, e.g.
`breeze --backend postgres --python 3.10 testing tests --test-type All`.
+UV workspace monorepo. Key paths:
-## Building documentation
+- `airflow-core/src/airflow/` — core scheduler, API, CLI, models
+ - `models/` — SQLAlchemy models (DAG, TaskInstance, DagRun, Asset, etc.)
+ - `jobs/` — scheduler, triggerer, DAG processor runners
+ - `api_fastapi/core_api/` — public REST API v2, UI endpoints
+ - `api_fastapi/execution_api/` — task execution communication API
+ - `dag_processing/` — DAG parsing and validation
Review Comment:
```suggestion
- `dag_processing/` — Dag parsing and validation
```
##########
AGENTS.md:
##########
@@ -3,29 +3,100 @@
# AGENTS instructions
-The main developer documentation lives in the `contributing-docs` directory.
The following points summarise
-how to set up the environment, run checks, build docs and follow the PR
workflow.
+## Environment Setup
-## Local virtualenv and Breeze
+- Install prek: `uv tool install prek`
+- Enable commit hooks: `prek install`
+- **Never run pytest, python, or airflow commands directly on the host** —
always use `breeze`.
+- Place temporary scripts in `dev/` (mounted as `/opt/airflow/dev/` inside
Breeze).
-- [`07_local_virtualenv.rst`](contributing-docs/07_local_virtualenv.rst)
explains how to prepare a local Python environment using `uv`. The tool creates
and syncs a `.venv` and installs dependencies with commands such as `uv venv`
and `uv sync`.
--
[`06_development_environments.rst`](contributing-docs/06_development_environments.rst)
compares the local virtualenv with the Docker based Breeze environment. Breeze
replicates CI and includes services like databases for integration tests.
+## Commands
-## Prek hooks
+- **Run a single test:** `breeze run pytest
path/to/test.py::TestClass::test_method -xvs`
+- **Run a test file:** `breeze run pytest path/to/test.py -xvs`
+- **Run a Python script:** `breeze run python dev/my_script.py`
+- **Run Airflow CLI:** `breeze run airflow dags list`
+- **Type-check:** `breeze run mypy path/to/code`
+- **Lint/format (runs on host):** `prek run --all-files`
+- **Lint with ruff only:** `prek run ruff --all-files`
+- **Format with ruff only:** `prek run ruff-format --all-files`
+- **Build docs:** `breeze build-docs`
-- Installation and usage of `prek` are described in
[`03a_contributors_quick_start_beginners.rst`](contributing-docs/03a_contributors_quick_start_beginners.rst).
Install with `uv tool install prek` and run checks via `prek --all-files`.
-- [`08_static_code_checks.rst`](contributing-docs/08_static_code_checks.rst)
provides more details on the available hooks and prerequisites. Enable the
hooks with `prek install` so they run automatically on each commit.
+SQLite is the default backend. Use `--backend postgres` or `--backend mysql`
for integration tests that need those databases. If Docker networking fails,
run `docker network prune`.
-## Running tests
+## Repository Structure
--
[`03a_contributors_quick_start_beginners.rst`](contributing-docs/03a_contributors_quick_start_beginners.rst)
shows running tests inside Breeze. Use `pytest` inside the container for
individual files or invoke `breeze testing` commands to run full suites, e.g.
`breeze --backend postgres --python 3.10 testing tests --test-type All`.
+UV workspace monorepo. Key paths:
-## Building documentation
+- `airflow-core/src/airflow/` — core scheduler, API, CLI, models
+ - `models/` — SQLAlchemy models (DAG, TaskInstance, DagRun, Asset, etc.)
+ - `jobs/` — scheduler, triggerer, DAG processor runners
Review Comment:
```suggestion
- `models/` — SQLAlchemy models (Dag, TaskInstance, DagRun, Asset, etc.)
- `jobs/` — scheduler, triggerer, Dag processor runners
```
Should we just use DagModel?
##########
AGENTS.md:
##########
@@ -3,29 +3,100 @@
# AGENTS instructions
-The main developer documentation lives in the `contributing-docs` directory.
The following points summarise
-how to set up the environment, run checks, build docs and follow the PR
workflow.
+## Environment Setup
-## Local virtualenv and Breeze
+- Install prek: `uv tool install prek`
+- Enable commit hooks: `prek install`
+- **Never run pytest, python, or airflow commands directly on the host** —
always use `breeze`.
+- Place temporary scripts in `dev/` (mounted as `/opt/airflow/dev/` inside
Breeze).
-- [`07_local_virtualenv.rst`](contributing-docs/07_local_virtualenv.rst)
explains how to prepare a local Python environment using `uv`. The tool creates
and syncs a `.venv` and installs dependencies with commands such as `uv venv`
and `uv sync`.
--
[`06_development_environments.rst`](contributing-docs/06_development_environments.rst)
compares the local virtualenv with the Docker based Breeze environment. Breeze
replicates CI and includes services like databases for integration tests.
+## Commands
-## Prek hooks
+- **Run a single test:** `breeze run pytest
path/to/test.py::TestClass::test_method -xvs`
+- **Run a test file:** `breeze run pytest path/to/test.py -xvs`
+- **Run a Python script:** `breeze run python dev/my_script.py`
+- **Run Airflow CLI:** `breeze run airflow dags list`
+- **Type-check:** `breeze run mypy path/to/code`
+- **Lint/format (runs on host):** `prek run --all-files`
+- **Lint with ruff only:** `prek run ruff --all-files`
+- **Format with ruff only:** `prek run ruff-format --all-files`
+- **Build docs:** `breeze build-docs`
-- Installation and usage of `prek` are described in
[`03a_contributors_quick_start_beginners.rst`](contributing-docs/03a_contributors_quick_start_beginners.rst).
Install with `uv tool install prek` and run checks via `prek --all-files`.
-- [`08_static_code_checks.rst`](contributing-docs/08_static_code_checks.rst)
provides more details on the available hooks and prerequisites. Enable the
hooks with `prek install` so they run automatically on each commit.
+SQLite is the default backend. Use `--backend postgres` or `--backend mysql`
for integration tests that need those databases. If Docker networking fails,
run `docker network prune`.
-## Running tests
+## Repository Structure
--
[`03a_contributors_quick_start_beginners.rst`](contributing-docs/03a_contributors_quick_start_beginners.rst)
shows running tests inside Breeze. Use `pytest` inside the container for
individual files or invoke `breeze testing` commands to run full suites, e.g.
`breeze --backend postgres --python 3.10 testing tests --test-type All`.
+UV workspace monorepo. Key paths:
-## Building documentation
+- `airflow-core/src/airflow/` — core scheduler, API, CLI, models
+ - `models/` — SQLAlchemy models (DAG, TaskInstance, DagRun, Asset, etc.)
+ - `jobs/` — scheduler, triggerer, DAG processor runners
+ - `api_fastapi/core_api/` — public REST API v2, UI endpoints
+ - `api_fastapi/execution_api/` — task execution communication API
+ - `dag_processing/` — DAG parsing and validation
+ - `cli/` — command-line interface
+ - `ui/` — React/TypeScript web interface (Vite)
+- `task-sdk/` — lightweight SDK for DAG authoring and task execution runtime
+ - `src/airflow/sdk/execution_time/` — task runner, supervisor
+- `providers/` — 100+ provider packages, each with its own `pyproject.toml`
+- `airflow-ctl/` — management CLI tool
+- `chart/` — Helm chart for Kubernetes deployment
-- Documentation can be built locally using `uv run --group docs build-docs` as
described in
[`11_documentation_building.rst`](contributing-docs/11_documentation_building.rst).
Within Breeze the equivalent command is `breeze build-docs`.
+## Architecture Boundaries
-## Pull request guidelines
+1. Users author DAGs with the Task SDK (`airflow.sdk`).
+2. DAG Processor parses DAG files in isolated processes and stores serialized
DAGs in the metadata DB.
+3. Scheduler reads serialized DAGs — **never runs user code** — and creates
DAG runs / task instances.
+4. Workers execute tasks via Task SDK and communicate with the API server
through the Execution API — **never access the metadata DB directly**.
+5. API Server serves the React UI and handles all client-database interactions.
+6. Triggerer evaluates deferred tasks/sensors in isolated processes.
-- Follow the PR guidance in
[`05_pull_requests.rst`](contributing-docs/05_pull_requests.rst). Always add
tests, keep your branch rebased instead of merged, and adhere to the commit
message recommendations from [cbea.ms/git-commit](https://cbea.ms/git-commit/).
+## Coding Standards
-For advanced topics such as packaging providers and API versioning see
[`12_provider_distributions.rst`](contributing-docs/12_provider_distributions.rst)
and
[`19_execution_api_versioning.rst`](contributing-docs/19_execution_api_versioning.rst).
+- No `assert` in production code.
+- `time.monotonic()` for durations, not `time.time()`.
+- In `airflow-core`, functions with a `session` parameter must not call
`session.commit()`. Use keyword-only `session` parameters.
+- Imports at top of file. Valid exceptions: circular imports, lazy loading for
worker isolation, `TYPE_CHECKING` blocks.
+- Guard heavy type-only imports (e.g., `kubernetes.client`) with
`TYPE_CHECKING` in multi-process code paths.
+- Apache License header on all new files (prek enforces this).
+
+## Testing Standards
+
+- Add tests for new behavior — cover success, failure, and edge cases.
+- Use pytest patterns, not `unittest.TestCase`.
+- Use `spec`/`autospec` when mocking.
+- Use `time_machine` for time-dependent tests.
+- Use `@pytest.mark.parametrize` for multiple similar inputs.
+- Test fixtures: `devel-common/src/tests_common/pytest_plugin.py`.
+- Test location mirrors source: `airflow/cli/cli_parser.py` →
`tests/cli/test_cli_parser.py`.
+
+## Commits and PRs
+
+Write commit messages focused on user impact, not implementation details.
+
+- **Good:** `Fix airflow dags test command failure without serialized DAGs`
Review Comment:
```suggestion
- **Good:** `Fix airflow dags test command failure without serialized Dags`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
