jason810496 opened a new pull request, #70671: URL: https://github.com/apache/airflow/pull/70671
**part of the streaming task log series** - related: #69299 (merged core plumbing), #69300 (merged KubernetesExecutor counterpart) ## Why `FileTaskHandler._read` wraps every read in `LogStreamAccumulator`, which eagerly drains the whole merged log stream (5000 lines to memory, the rest to a tempfile on the API server) *before the first byte reaches the client* — only to get the `total_lines` for the continuation token. For a finished task no continuation token is ever issued (`end_of_log=True` on both the JSON and NDJSON paths), so the count is dead weight: every terminal-task read full-log downloads — pays a full-log disk write + read-back + re-parse and a time-to-first-byte equal to the whole drain, for nothing. ## What - Early-return in `FileTaskHandler._read` bypassing `LogStreamAccumulator` when `end_of_log` is `True`. - **Memory stays bounded either way** — the k-way merge heap in `_interleave_logs` (capped at `HEAP_DUMP_SIZE`, flushing half when full) is what bounds the read path, not the accumulator. --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes, with help of Claude Code Sonnet 5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
