potiuk commented on issue #37266: URL: https://github.com/apache/airflow/issues/37266#issuecomment-1935091821
You can already do that by implementing cusom handlers via https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html. We are using standard python logging functionality and even added a way to configure loggers via dictionary. If you want to log things remotely, you can find python logging handler that does and plug them in that advanced configuration - no problem with that. I see no particular reason why Airflow would have to add additional layer of abstraction there and be opinionated about your logging choice where standard Python logging framework is fully extensible, has a number of handlers implemented (including Rotating handlers). There are a number of handlers implemented already that can do any kind of logging remotely. Pretty much every cloud storage provider or logging system provider comes with a python logging handler that you can just plug-in (and custom configuration of ours allows you to plug them in in whatever way you think makes sense - using standard hierarchy of loggers based on python packages. The only reason we have `logging task handlers` and `remote logging` for tasks is that they need to be integrated in Airflow UI and by those remote handlers we can not only push stuff to S3 but also pull it for the UI purposse I do not think we should add something extra here, rather than that I'd focus on adding Open Telemetry logging integration which we already agreed to as an AIP (https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow) . This is much better abstractiion (and one that we do not have to define and has a chance to be well standardized) and out-of-the-box has support for pretty much anything. Converting it into a discussion if more is needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
