virendhar-aws opened a new issue #22512: URL: https://github.com/apache/airflow/issues/22512
### Description # Objective: Add functionality to Airflow ECSOperator to capture the task execution logs as they happen from AWS CloudWatch to the ECSOperator's task execution logs. ### Use case/motivation Present Challenge: When executing Fargate tasks via ECSOperator the logs are not readily available in Airflow task during execution. The logs are shown after execution and final status of the container execution is reflected in Airflow ECS Operator task logs. As a result one should navigate the task logs in Airflow to get a handle of the container log path in AWS Cloudwatch and then access the log from AWS Cloudwatch in realtime. The current behavior introduces the following challenges, The ongoing execution logs are not captured in Airflow task ( ECS Operator ) log. This typically involves couple of additional steps - which can be avoided. Also for long execution tasks, one has to wait till the task is executed to debug issues if any # Solution Mechanism to relay the logs from AWS CloudWatch during execution to Airflow logs as and when they are happening in realtime (or near realtime < 2 mins ). ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
