mtraynham opened a new pull request, #64481: URL: https://github.com/apache/airflow/pull/64481
## Summary - Adds a new `[logging] task_log_to_stdout` boolean config option (default `False`) that forwards task subprocess log messages to the worker's stdout, in addition to the task log file and any configured remote log handler. - Passes the config value as `subprocess_logs_to_stdout` in both `LocalExecutor` and `CeleryExecutor` call sites to `supervise()`. ## Motivation In Airflow 3, tasks run in a forked subprocess managed by `WatchedSubprocess`. The subprocess uses structlog over a JSON socket channel back to the supervisor, which writes logs only to the task log file by default. The user's `LOGGING_CONFIG` dict (via `AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS`) is never applied inside the task subprocess, so the Airflow 2 pattern of adding a `StreamHandler` to the `airflow.task` logger no longer works (https://stackoverflow.com/a/52261504). The `subprocess_logs_to_stdout` flag already exists on `WatchedSubprocess` and `supervise()`, but defaults to `False` and is not exposed as a config option. Neither `CeleryExecutor` nor `LocalExecutor` passes `True`. There is no way for users to enable it without monkey-patching. Container-based deployments (ECS, Kubernetes, Cloud Run, etc.) commonly rely on stdout-based log collection (CloudWatch, Fluentd, Fluent Bit, Datadog, etc.). Without this option, task execution logs are invisible to these collectors. In case of an existing issue, reference it using one of the following: * related: #49863 * related: #54501 * related: #58053 --- ##### Was generative AI tooling used to co-author this PR? - [ ] Yes (please specify the tool below) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
