jason810496 commented on code in PR #50715:
URL: https://github.com/apache/airflow/pull/50715#discussion_r2101658101
##########
airflow-core/src/airflow/utils/log/log_reader.py:
##########
@@ -96,10 +99,15 @@ def read_log_stream(self, ti: TaskInstance, try_number: int
| None, metadata: di
TaskInstanceState.RUNNING,
TaskInstanceState.DEFERRED,
):
- if not logs[0]:
+ if not logs:
# we did not receive any logs in this loop
# sleeping to conserve resources / limit requests on
external services
time.sleep(self.STREAM_LOOP_SLEEP_SECONDS)
+ empty_iterations += 1
Review Comment:
I think this scenario occurs when the current `TaskInstance` is still
running, but it's a long-running task.
1. The current `end_of_log` will still be `true` if the `TaskInstance` is
still `running` or `deferred`:
https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/file_task_handler.py#L446C9-L449
2. `FileTaskHandler` will skip the same logs (based on `log_pos` in
metadata), so if the `logs` returned from `read_log_chunks` remain empty for
several iterations, we can infer that the `TaskInstance` is still running but
not producing new logs (still polling or computing...).
https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/file_task_handler.py#L451-L452
This behavior causes the following error:
https://github.com/apache/airflow/issues/39323#issuecomment-2850639129
Therefore, we need to check for multiple consecutive empty iterations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
