Admaing opened a new pull request, #73266:
URL: https://github.com/apache/airflow/pull/73266
The quick-start kept every task log forever, so the shared `logs` volume
grows without bound until the
disk of the machine running it fills up.
The Helm chart has had a `logGroomerSidecar` for a long time, but the
`docker-compose.yaml` that most
people start with had no equivalent, and the logging docs never said that
Airflow neither rotates nor
deletes local task logs.
This adds an `airflow-log-groomer` service that reuses the same
`/clean-logs` script as the chart:
* task logs older than 15 days are deleted by default (the same default as
the Helm chart), configurable
with `AIRFLOW__LOG_RETENTION_DAYS` / `AIRFLOW__LOG_RETENTION_MINUTES`;
* `AIRFLOW__LOG_MAX_SIZE_BYTES` / `AIRFLOW__LOG_MAX_SIZE_PERCENT` bound the
size of the logs folder;
* removing the `airflow-log-groomer` service keeps all logs, as before;
* the behaviour is documented in the quick-start page, in "Logging for
Tasks", and in the advanced
logging configuration page, which previously only mentioned size-based
rotation.
Note for reviewers: the `AIRFLOW__LOG_*` variables are read by the
`/clean-logs` script, not by Airflow
itself. Setting them in `.env` therefore adds them to the environment of
every service, where Airflow
ignores them (`AIRFLOW__LOG_RETENTION_DAYS` has no `__` separator between a
section and a key, so it is
not a configuration option), while the groomer picks them up through Compose
interpolation. The names
are the ones the chart and the script already use.
Tested with `docker compose config` — the rendered output was checked for
both the default and an
overridden retention — plus the new `test_log_groomer_service_config` test,
which asserts the service,
its command, and the retention default/override. `yamllint`, the Compose
JSON schema hook, mypy for
`docker-tests`, and the remaining prek hooks pass for the changed files.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: DeepSeek Harness (deepseek-v4.1-flash) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
