raghavan-arvind opened a new pull request, #71567: URL: https://github.com/apache/airflow/pull/71567
## Why In deferrable mode, when the trigger fires its terminal event, `trigger_reentry` calls `_write_logs`, which re-emits the **entire pod log** into the task log in one bulk fetch. For chatty workloads this is problematic: - Most production setups already ship pod stdout to a log aggregator directly from the node; the bulk replay double-ingests every line (and stores a second copy in remote task logs). - The "full log" is illusory anyway: the kubelet rotates container logs (`containerLogMaxSize`, default 10Mi), and the pod logs API only serves the current file — so long-running chatty pods already get a silent, size-based, node-config-dependent tail today. An explicit, predictable line-based cap is strictly more honest than the status quo, and lets the task log serve its real purpose: showing how the job ended. ## What Adds `on_deferrable_finish_log_tail_lines: int | None = None` to `KubernetesPodOperator`: - `None` (default) preserves existing behavior exactly. - When set, `_write_logs` passes `tail_lines` through to `read_namespaced_pod_log`, so only the last N lines are written to the task log in the single bulk fetch after the pod finishes. - Deliberately scoped to that one fetch: it does **not** affect live log streaming (non-deferrable mode) or the periodic `logging_interval` relay, where tailing each window would silently drop lines. `_write_logs` takes `tail_lines` as a plain parameter, so call sites stay explicit. - When `logging_interval` is set, the finish-time fetch already passes `since_time`; the Kubernetes API applies both (time window first, tail within it), which keeps the semantics consistent. Happy to adjust the parameter name if reviewers prefer something shorter. ## Testing Verified in a production-like staging deployment (Airflow 3.3.0, provider vendored with this patch): a deferrable pod task with `on_deferrable_finish_log_tail_lines=5` wrote exactly the last 5 lines of the container log after resume, with defer/resume/XCom/cleanup behavior unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
