brki commented on issue #53442: URL: https://github.com/apache/airflow/issues/53442#issuecomment-3486658535
I would also very much like to be able to add some custom log processor. My use case, that was working fine with Airflow 2, by adding a custom handler: I want to collect task log lines and periodically write them in a chunk to a custom log file. I've done this on Airflow 2, because 1) I want to be able to see these log chunks as multiline log entries in Splunk, and 2) because the otel collector we use, with a modest amount of CPU and memory, chokes when it's configured to monitor all the individual standard task logs under AIRFLOW_HOME/logs/dag_id=WHATEVER/task_id=WHATEVER/*.log . I don't see any way to do this in Airflow 3. Perhaps there is, I don't know. Can we somehow just inject an additional processor to the structlog processors for all task logs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
