Steven Reitsma created AIRFLOW-5027:
---------------------------------------
Summary: Generalized CloudWatch log grabbing for ECS and SageMaker
operators
Key: AIRFLOW-5027
URL: https://issues.apache.org/jira/browse/AIRFLOW-5027
Project: Apache Airflow
Issue Type: Improvement
Components: aws, contrib, logging
Affects Versions: 1.10.3, 1.10.4
Reporter: Steven Reitsma
Assignee: Steven Reitsma
It's very useful to have the CloudWatch task logs of an ECS/SageMaker job
available within Airflow. Currently, this is not possible for ECS tasks.
The SageMaker operator starts a job on AWS SageMaker. It already supports
grabbing the CloudWatch logs of the (finished) job to the Airflow instance.
The ECS operator is very similar in that it starts a job on AWS ECS. It
doesn't, however, support grabbing CloudWatch logs at the end of a job.
I've separated the existing log grabbing of the SageMaker operator to a hook
and both the SageMaker and ECS operators now call that hook at the end of a job.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
