kahlstrm commented on issue #41164: URL: https://github.com/apache/airflow/issues/41164#issuecomment-2261742577
> @kahlstrm / @RNHTTR > > Can you speak more about the change introduced in #39177 in case my interpretation is insufficient / incorrect. > The webserver only reads from the worker / triggerer log server when there are no local logs or remote logs. This behaviour was introduced in https://github.com/apache/airflow/pull/39177. To clarify on this point, #39177 introduced this particular behaviour as an alternative implementation of #32561, which entirely removed fetching logs from the worker / triggerer log server for past task runs. The way #39177 was implemented is to retain the wanted behaviour of #32561 of not triggering the HTTP request in cases where remote logs were found, but still to support our use case of storing the logs on the worker with a persistent volume. The deferred state logic was kept to be as close as possible to the previous implementations, however it became evident in https://github.com/apache/airflow/pull/39496#issuecomment-2149692239, adding tests for the deferred state caused test flakiness with unexpected results. It might be that this has caused a regression in the deferred state, as that one was untested in our use case, whereas viewing previous task attempts was confirmed to be working as expected again after #39177. I would be totally fine going back to behavior prior to #32561, as I that initially introduced this behavior of not serving the worker / triggerer logs in certain circumstances. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
