vandonr-amz commented on code in PR #31824:
URL: https://github.com/apache/airflow/pull/31824#discussion_r1225596631
##########
airflow/providers/amazon/aws/hooks/logs.py:
##########
@@ -113,16 +113,16 @@ def get_log_events(
yield from events
+ if continuation_token.value == response["nextForwardToken"]:
+ return
+
if not event_count:
num_consecutive_empty_response += 1
if num_consecutive_empty_response >=
NUM_CONSECUTIVE_EMPTY_RESPONSE_EXIT_THRESHOLD:
# Exit if there are more than
NUM_CONSECUTIVE_EMPTY_RESPONSE_EXIT_THRESHOLD consecutive
# empty responses
return
- elif continuation_token.value != response["nextForwardToken"]:
- num_consecutive_empty_response = 0
Review Comment:
doing this check after the check on `not event_count` means that even when
we get the same token twice in a row, clearly indicating that there is nothing
more to read, we still do `NUM_CONSECUTIVE_EMPTY_RESPONSE_EXIT_THRESHOLD` (=3)
calls before returning.
Moving the check on token above saves 2 calls every time we get the correct
termination condition from cloudwatch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
