ferruzzi commented on code in PR #70635:
URL: https://github.com/apache/airflow/pull/70635#discussion_r3668790812
##########
providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py:
##########
@@ -271,8 +272,64 @@ def stream(self, relative_path: str, ti: RuntimeTI) ->
StreamingLogResponse:
except Exception as e:
messages.append(str(e))
+ # A deferred task's triggerer logs are stored under a *separate* stream
+ # (`<task log stream>.trigger.<job id>.log`) rather than the task's
own stream --
+ # see FileTaskHandler.add_triggerer_suffix / TriggerLoggingFactory.
The local-file
+ # reader picks these up for free via `glob(worker_log_path.name + "*")`
+ # (FileTaskHandler._read_from_local); CloudWatch has no glob
equivalent, so they
+ # have to be discovered explicitly via a ListLogStreams-style prefix
scan.
+ #
+ # Skip this while the task is actively DEFERRED: the UI already tails
those logs
+ # live from the triggerer over HTTP in that state (FileTaskHandler
+ # ._read_from_logs_server), so the extra CloudWatch API call would be
both redundant
+ # and, for a long-running deferral, repeated on every UI poll.
Review Comment:
This may be an edge case, but what happens if the task is deferred again?
`_get_log_retrieval_url` builds the served-log URL from `ti.triggerer_job.id`
so the UI is only tailing the logs of the current deferred task. Do we need
some way to persist the logs from previous deferrals into the main logs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
