kahlstrm commented on PR #39177: URL: https://github.com/apache/airflow/pull/39177#issuecomment-2270062385
> @kahlstrm can you clarify what you mean here? > > > This PR implements #32561 in a different way. This caused a regression for our use case, where non-running task try logs weren't shown in UI for running tasks. This is due to us storing the logs on the worker with a Persistent Volume. > > Specifically this part: > > > This is due to us storing the logs on the worker with a Persistent Volume > > What does storing logs on the worker with a PV have to do with anything? If you're storing logs on a PV, shouldn't the webserver have access to it, so it can read the logs directly from the PV? > > This PR definitely has introduced a bug, because now users cannot see served logs from triggerer while deferred. But I'm just not sure exactly what functionality here we need to preserve and implement in a different way. I'm no longer working with this particular project, but the setup was a PV on the worker that was not mounted on the webserver. When it comes to the bug, I would guess [this line change](https://github.com/apache/airflow/pull/39177/files#diff-e7f34f73940eb52d92bb991abedc1c963431c5373c12dff739c8fb7d03e93d3aR384) is the culprit for the behavior. The reasoning for that line was to enable fetching previous task instance attempt served logs when there are no remote logs available, but this then introduced the incorrect behavior for the deferred case. Is `TaskInstanceState.DEFERRED` always the latest task task instance attempt? If yes, then changing the aforementioned line to the following would perhaps fix this: ```python if is_in_running_or_deferred and not executor_messages and (not remote_logs or ti.try_number == try_number): ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
