potiuk commented on issue #36972: URL: https://github.com/apache/airflow/issues/36972#issuecomment-1906724357
Actually I am not sure if we want to do it at all. I don't see who could benefit from that. Not our users, I am quite sure of that. For users it's better to focus on what has already been approved in https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow - i.e. open-telemetry logging integration. This is waiting for someone to implement it and if someone was to focus on improving our logging, that would be it. What we want to achieve is that none of our users shoud focus on writing their own Handlers, they should simply configure the right OpenTelemetry integration if they want to do something with logs. And if someone would like spend their effort there, that would be way more efficient way of improving lifes of our users. Contributors and Maintainers? I doubt the effort and cost needed would be justified by the benefits. I believe that framework could potentially make live a bit easier for maintainers of Airflow, but sinve we already have working implementation of almost-declarative way of configuring logging and handlers, and we've already made a lot of effort into not only making our handlers task-aware and also being able to log things to memory for Triggerer (asyncio) and to merge the logs coming from diferent context from diffferent contexts. Not mentioning the fact that just basic task handlers of Airflow already support much more than regular handlers - they not only allow to write logs to local or remote storage, but also allow to read the logs written for the webserver to retrieve them via API from workers that produced them so that they can show up in a UI of a webserver running on a completely diffierent machine. Not mentioning filters (which loguru mentions as not needed) that we are using for secrets masking and our automated way of coloring logs. Not even mentioning the custom handlers we have that are handling the cases that in POSIX system by default writing to log files by default fills up the filesyste, cache and (even if it is not really causing memory issues, only scares away some people) we have custom handlers that write logs to files while avoiding to fill up the cache. Not mentioning that we are doing custom redirections of stdin and stderr from subprocesses and redirecting them back to log handler in order to handle the situation when we want to capture all the stdout/stderr printed and prentend it has been logged using context-aware logger. Simply the scope of "logging" in Airflow and amount of engineering efforts spent in that is so huge, and the amount of code you'd need to rewrite is so big and prone to errors that it's a huge effort to be able to use something like loguru. But if @lucasfcnunes, you feel like creating a pull request for that is a good idea, then you can always try - maybe I am exaggerating the effort needed, after just looking at the demo and docs of loguru, who knows. Converting it into discussion anyway, because maybe indeed, I am wrong and there are benefits that will outhweigh the cost needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
