tggm803sc opened a new pull request, #77077:
URL: https://github.com/apache/airflow/pull/77077
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
Adds a second Docker Compose example that runs Airflow with `LocalExecutor`
and PostgreSQL, plus
a short docs page explaining when that is a better fit than the
CeleryExecutor quick start.
The existing quick start needs Redis, a Celery worker and optionally Flower.
For trying Airflow,
developing Dags, or a small single-machine installation none of that is
required: `LocalExecutor`
runs tasks in the scheduler's own parallel subprocesses, so PostgreSQL is
the only extra container.
related: #16031
### What changed
- `airflow-core/docs/howto/docker-compose/docker-compose-localexecutor.yaml`
— the example.
Services: `postgres`, `airflow-apiserver`, `airflow-scheduler`,
`airflow-dag-processor`,
`airflow-triggerer`, `airflow-init`, plus the optional `airflow-cli` debug
service.
No Redis, no `airflow-worker`, no Flower.
- `airflow-core/docs/howto/docker-compose/local-executor.rst` — when
LocalExecutor is the right
choice, initialize/run/cleanup, a note that there is no worker container
so task logs come from
the scheduler container, and cross-links to the CeleryExecutor guide and
the LocalExecutor
concept page.
This takes the "more examples" route rather than the generator/wizard
discussed on the issue,
following the earlier comment that a wizard is long-term work.
**Deliberately `related:` rather than `closes:`** — the issue also mentions
other configurations
(e.g. MySQL). This PR adds one example to keep review focused; I am happy to
follow up with more.
### Why FabAuthManager
The file sets `AIRFLOW__CORE__AUTH_MANAGER` to `FabAuthManager`, matching
the existing
CeleryExecutor example. In Airflow 3 SimpleAuthManager is the default and
manages users through
configuration, so `_AIRFLOW_WWW_USER_CREATE` / `_AIRFLOW_WWW_USER_USERNAME`
silently create
nothing and the documented `airflow`/`airflow` login would not exist. FAB
keeps the init
container's account creation working and keeps the two examples consistent.
The trade-off is that this example does not demonstrate the 3.x default. If
you would rather it
showcased SimpleAuthManager, the alternative is to drop the auth-manager
override and document
that the generated password is written to
`${AIRFLOW_HOME}/simple_auth_manager_passwords.json.generated`. I am happy
to switch.
### Testing
Static checks, plus the parts of the file that can be exercised without
starting containers, run
against `apache-airflow==3.4.0` on Python 3.12 in a clean virtualenv:
- `docker compose config -q` — passes, including anchor and merge-key
resolution.
- `yamllint` with default rules — passes. Line length is scoped out at the
top of the file because
the auth-manager class path and the health-check values cannot be wrapped;
I can convert that to
per-block `disable`/`enable` pairs if that is the preferred style.
- All internal `:doc:` targets checked against `main`.
- **Every service command exists** in 3.4.0: `airflow api-server`, `airflow
scheduler`,
`airflow dag-processor`, `airflow triggerer`, `airflow db migrate`,
`airflow jobs check`,
`airflow dags list`, `airflow dags test`.
- **Every `AIRFLOW__*` variable in the file resolves to a real option** (via
`airflow config
get-value`), including `core.execution_api_server_url`,
`api_auth.jwt_secret` and
`scheduler.enable_health_check`.
- `airflow db migrate` — exit 0, "Database migration done!".
- The auth-manager claim, verified both ways: with
`apache-airflow-providers-fab` installed and
`auth_manager` set to FabAuthManager, `airflow users create` succeeds and
`airflow users list`
shows the account. Without the provider there is no `airflow users`
command at all (exit 2,
invalid choice), and the image entrypoint skips account creation
explicitly:
```
Skipping user creation as auth manager different from Fab is used
```
The repo `Dockerfile` includes `fab` in `AIRFLOW_EXTRAS`, so the official
image ships the
provider this file depends on.
- **Still outstanding:** the containers themselves have not been started.
The machine I could test
on has no Docker daemon access, so `docker compose up`, the
`airflow`/`airflow` login and a task
actually executing under LocalExecutor remain unverified. I will run those
on a Docker host with
at least 4 GB and update this section before marking the PR ready for
review, so it stays in
Draft until then.
- Sphinx docs build not run locally.
### Questions for reviewers
1. Is a second YAML in `airflow-core/docs/howto/docker-compose/` the right
home? I want to confirm
an extra YAML there is published the same way `docker-compose.yaml` is,
so the new page can
offer a `curl` download from the docs URL. If that directory should hold
only the quick-start
file, what would you prefer instead?
2. FAB vs SimpleAuthManager, as described above.
3. One example in this PR, or a small set?
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Codebuff following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
Note: commit author/co-author name and email in commits become permanently
public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
(Not added here: this is a docs example, not a behaviour change — say if
you want one.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
