potiuk commented on issue #59634: URL: https://github.com/apache/airflow/issues/59634#issuecomment-3687700431
If that is really what happens, I think that is indeed somewhat loss of functionality - especially for logs coming from 3rd-party dependencies. While we can probably expect people to use LoggingMixin for Dag code, this is not realistic to expect them to do any kind of complex custom logging configuration to see the logs in task logs. That would be - I think an unwelcome regression from Airfflow 2. I am not sure if that is from the logging level, or (more likely) with the need of capturing the logs and sending them via supervisor and the fact that it does not happen by default. And whether it is easy to fix now - do not know that much about structlog, but I think w nhen airflow logging level should be set to DEBUG, the task log should provide logs from both - airflow logs and 3rd-party libraries. You might want to configure logging separately in "advanced" configuration - but I think it requires quite a lot of internal knowledge how airflow logging works - and even if we decide DEBUG setting should only be working for Airflow, we need to make it easy to setup and describe. Similarly if a library does "log.info()" from any logger, there is no particular reason why this log should not be visible in task logs by default - this is natural and intuitive expectation of our users, and we seem to have taken away that from them now. @ashb @amoghrajesh @kaxil - WDYT? Was it a deliberate choice we made when designing the task isolation work, or simply a byproduct of switching to structlog and task isolation? It looks a bit like the latter, but maybe there were some good reasons why we made that decision and some easy ways to bring back the "intuitive" behaviour? I think this is pretty important to have answer to before we decide what we do with it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
