o-nikolas commented on issue #43717: URL: https://github.com/apache/airflow/issues/43717#issuecomment-2704992003
> Hello [@o-nikolas](https://github.com/o-nikolas), regarding this question: > > > Is the issue only occurring with this image? Do you use other images which do not see this behaviour? Also, is it always the tail end of the logs that is missing? Or is it ever logs at the beginning or middle of the stream? > > I need to refresh my memory on how this impacts ECS tasks other than with our image executing DBT, but the logs in the original issue are always truncated, never missing from the middle or beginning of the task. Thanks for confirming this! And if you do get around to seeing if any of your other images are affected, that would be greatly appreciated :) > > My belief is that this is caused by logs being pushed to the same timestamp in the logstream, and subsequently not being pulled because the timestamp is misaligned with the actual execution time. This should be visible in the logstreams attached for the successful task execution. Interesting, we already had a fix in place for this hypothesis: https://github.com/apache/airflow/pull/41515 I wonder if a longer sleep is required there, or perhaps an altogether different fix. My suspicion is that the issue is eventual consistency on the cloudwatch side. I.e. those final logs aren't available to be read yet by the time we attempt to pull them. But more investigation is required. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
