potiuk commented on issue #32379: URL: https://github.com/apache/airflow/issues/32379#issuecomment-1657199805
I am not sure if I can assess it by looking at the details. but yes is it sounds about right, though it might terribly complicate "serve log" handing (it is already terribly complicated as @dstandish might attest). Each of those steps (for example where to display it in the UI is a separate decision on it's own - like how to make it UX-friendly currently, should we surface it somewhere ? It seems like awful lot of complexity for very little value, I think we should answer a question - why would someone want to see the callbacllogs in the UI? When you surface task logs, then yeah - it makes sense to see what's going on, but seeing callback logs is marginally useful, I think. But maybe it is worth it when it is paired with another change - related to your question: > A specific DagRun may run many times, so we need to keep track of how many times it ran and need to show logs of each attempt of the callback. Currently, are we tracking how many times a dag run ran? I **believe** even today we have a problem tha we do not store hostname of the previous run attempts - only the last one - and it causes problems with retrieving the logs from previious runs if there is no remote logging configured and no log persistence. Maybe that is an opportunity to fix it as part of this issue. I still would love to hear from @dstandish what he thinks - he has been recently "knee-deep" or rather "waist-deep" in the logging part, and knows a bit of the complexities involved, so maybe the judgement of Daniel on whether it's worth to bring in the callback logs might help with the decisions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
