luc-pimentel opened a new pull request, #74165: URL: https://github.com/apache/airflow/pull/74165
With `[logging] dag_processor_log_target = stdout`, the Dag processor still writes every parse's output to `logs/dag_processor/<date>/<bundle>/<file>.log`, and since #65528 it also prints the same lines to stdout. Nothing reads or removes those files, so the directory keeps growing. The option's description presents `file` and `stdout` as alternatives, and in Airflow 2 `stdout` replaced the file handler instead of adding to it. For `stdout`, `DagFileProcessorManager._get_logger_for_dag_file` now returns a stdout logger tagged with `dag_file` and `bundle_name` instead of opening a file. That logger is the parse process's own logger, so the child's output and the supervisor's messages about the parse (such as "Process exited abnormally", which only went to the file before) reach stdout once, in the same format as #65528's output. Nothing copies to stdout a second time anymore, so the `_get_target_loggers` override that tagged that copy is removed, and its tests are replaced by one checking that the stdout logger carries the tags. `file` mode is unchanged, and the option's description now says that `stdout` writes no log files. Tested on 3.3.2 and main with `airflow dag-processor -n 3`, using a Dag that prints one line and logs one warning when parsed (3 parses): | `dag_processor_log_target` | Before | With this change | | --- | --- | --- | | `file` | 1 log file with 6 parse lines, none on stdout | same | | `stdout` | 1 log file with 6 parse lines, the same 6 on stdout | no log directory, 6 parse lines on stdout | The new tests fail without this change. #71052 tried the same fix and was closed under the open PR limit. If you'd rather keep `stdout` writing the file too, I can put this behind a new `stdout_only` value instead. closes: #70968 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
