rchui opened a new issue, #31675: URL: https://github.com/apache/airflow/issues/31675
### Description Add functionality to Airflow BatchOperator to capture the task execution logs as they happen from AWS CloudWatch to the BatchOperator's task execution logs. ### Use case/motivation When executing Batch tasks via BatchOperator the logs are not readily available in Airflow task during execution; only a link is provided to point to where the logs can be viewed in CloudWatch. As a result one should navigate the task logs in Airflow to get a handle of the container log path in AWS Cloudwatch and then access the log from AWS Cloudwatch in realtime. The current behavior introduces the following challenge: The ongoing execution logs are not captured in Airflow task ( BatchOperator ) log. This typically involves couple of additional steps - which can be avoided. ### Related issues This is almost identical to the ECSOperator realtime log streaming that was raised in this issue: https://github.com/apache/airflow/issues/22512. In fact most of the description, use case, and motivation are lifted from that issue. An `EcsTaskLogFetcher` class already exists in `airflow.providers.amazon.aws.hooks.ecs` which AWS Batch can use as a backend. It just needs to be initialized, started, and stopped at the correct time. ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
