sunank200 commented on code in PR #50715:
URL: https://github.com/apache/airflow/pull/50715#discussion_r2101061394
##########
airflow-core/src/airflow/utils/log/log_reader.py:
##########
@@ -96,10 +99,15 @@ def read_log_stream(self, ti: TaskInstance, try_number: int
| None, metadata: di
TaskInstanceState.RUNNING,
TaskInstanceState.DEFERRED,
):
- if not logs[0]:
+ if not logs:
# we did not receive any logs in this loop
# sleeping to conserve resources / limit requests on
external services
time.sleep(self.STREAM_LOOP_SLEEP_SECONDS)
+ empty_iterations += 1
Review Comment:
Right now, it accumulates empties across the entire stream, even if some
non-empty batches slip in. For example, if it ever gets “hello”, then 2
empties, then another log line, you’ll still stop after the 5th empty overall.
It might be good to consider resetting `empty_iterations = 0` whenever it
actually receives a non-empty batch, to make it a counter of consecutive empty
polls.
##########
airflow-core/src/airflow/utils/log/log_reader.py:
##########
@@ -96,10 +99,15 @@ def read_log_stream(self, ti: TaskInstance, try_number: int
| None, metadata: di
TaskInstanceState.RUNNING,
TaskInstanceState.DEFERRED,
):
- if not logs[0]:
+ if not logs:
# we did not receive any logs in this loop
# sleeping to conserve resources / limit requests on
external services
time.sleep(self.STREAM_LOOP_SLEEP_SECONDS)
+ empty_iterations += 1
Review Comment:
Maybe something like the following at line 97:
```
if logs:
empty_iterations = 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
