jason810496 opened a new pull request, #69299:
URL: https://github.com/apache/airflow/pull/69299
- related:
1. This PR. **Merge first** - the other two build on it.
2. `refactor/logging/kubernetes-executor-streaming-logs` - implements
`get_streaming_task_log` for the KubernetesExecutor family. Depends on this PR.
3. `refactor/logging/configurable-running-pod-log-lines` - makes
`RUNNING_POD_LOG_LINES` configurable. Independent of 2 (no textual conflict),
can merge in any order after this PR.
## Why
When the API server reads logs for a RUNNING task, the executor's
`get_task_log` returns the whole log fully materialized, so it sits on the heap
at once before the bounded `LogStreamAccumulator` downstream (5000 messages
resident, rest spilled to disk) can do its job. Large logs spike the API
server's anonymous heap and can OOM it.
## What
- Add `get_streaming_task_log` and a `supports_streaming_logs` class
attribute (default `False`) to `BaseExecutor`.
- `FileTaskHandler._read` prefers the streaming method when the executor
advertises `supports_streaming_logs`, and falls back to the legacy
`get_task_log` otherwise, so provider and custom executors that haven't
implemented it keep working unchanged.
- No executor in this PR advertises support yet; the KubernetesExecutor
family implementation is the follow-up PR.
## Benchmark
A/B measurement of API server memory serving the same ~415 MB ndjson log (1M
lines) of a RUNNING KubernetesExecutor task through `GET .../logs/{try_number}`
with `Accept: application/x-ndjson`, sampling the API server cgroup. A =
materializing read (before this series), B = streaming read (this PR + the
KubernetesExecutor follow-up).
| Metric (API server cgroup) | A: materializing | B: streaming |
| --- | --- | --- |
| Peak anonymous heap growth | +2093.9 MiB | +179.9 MiB (~11.6x lower) |
| Peak RSS (`memory.current`) | 2964.4 MiB | 1193.4 MiB |
| Elapsed | 33.2 s | 19.4 s |
Without streaming the full log lives on the heap at once (~2.1 GiB
anonymous, non-reclaimable memory, which is what OOMs the API server). With
streaming the executor yields into `LogStreamAccumulator`; B's remaining RSS
growth is mostly reclaimable page cache from the accumulator's disk spill.
**Interpretation caveat:** `KubernetesExecutor.RUNNING_POD_LOG_LINES = 100`
normally caps the executor read to the last 100 lines, which would hide the
difference; the benchmark lifted the cap to 100,000,000 in both builds.
Production keeps the cap at 100, so this measures the mechanism's headroom for
executors returning large logs, not a shipped memory reduction today (the
`running_pod_log_lines` config PR is what makes the cap tunable). Single run
per build, and only the ndjson path streams end to end (`Accept:
application/json` buffers the whole response regardless).
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes, with help of Claude Code Fable 5 following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
