pyrr opened a new issue, #43130:
URL: https://github.com/apache/airflow/issues/43130
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.29.0
### Apache Airflow version
2.9.2
### Operating System
Amazon Linux 2023.5.20240805
### Deployment
Amazon (AWS) MWAA
### Deployment details
Using a Python 3.11 environment with botocore==1.34.106 and
apache-airflow-providers-amazon==8.29.0.
### What happened
The code for EcsRunTaskOperator looks for log streams by using the pattern
`prefix-name/ecs-task-id`:
```python
def _get_logs_stream_name(self) -> str:
return
f"{self.awslogs_stream_prefix}/{self._get_ecs_task_id(self.arn)}"
```
In the [AWS ECS docs for
LogConfiguration](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_LogConfiguration.html),
log streams are created using the pattern
`prefix-name/container-name/ecs-task-id`.
### What you think should happen instead
EcsRunTaskOperator should follow the same log stream naming convention as
specified in the AWS ECS docs.
Because this isn't the case, you need to specify a stream prefix using a
different pattern when registering your task vs. when running your task.
### How to reproduce
1. Register a task definition with a `containerDefinition` containing at
least the following:
```json
"name": "my-container-name",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "my-log-group",
"awslogs-region": "us-east-1",
"awslogs-create-group": "true",
"awslogs-stream-prefix": "my-container-prefix",
},
},
```
The logs in your ECS task will be uploaded to CloudWatch under the log group
`my-log-group` and the prefix `my-container-prefix/my-container-name/task-id`.
2. Run the task using the EcsRunTaskOperator with the same
`awslogs-stream-prefix`.
3. View the logs in Airflow-- they will fail to fetch the task logs from
CloudWatch because they are looking for the log group `my-log-group` and prefix
`my-container-prefix/task-id`.
### Anything else
The only workaround for this issue is to specify the task definition's
`awslogs-stream-prefix` with `my-prefix-name` and the EcsRunTaskOperator's
`awslogs_stream_prefix` with `my-prefix-name/my-container-name`.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
