fpiped opened a new issue, #74699:
URL: https://github.com/apache/airflow/issues/74699
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.2
### What happened and how to reproduce it?
When an ECS task's container exits non-zero and `EcsRunTaskOperator` cannot
find the task's CloudWatch log stream, the operator fails with CloudWatch's
`ResourceNotFoundException` instead of its own exception. `skip_on_exit_code`
is then ignored: a task that should be skipped fails.
The failure path builds the error message from the last log lines:
`_check_success_task` calls `AwsTaskLogFetcher.get_last_log_messages`, which
calls `GetLogEvents` without handling a missing stream. The fetcher's polling
loop handles the same case (`_get_log_events` catches
`ResourceNotFoundException` and logs "Cannot find log stream yet").
Reproduced on Fargate with `apache-airflow-providers-amazon` 9.37.0 (the
code is the same on `main`). The container runs `sh -c 'exit 3'`; the operator
has `skip_on_exit_code=[3]`, `awslogs_group` set and an `awslogs_stream_prefix`
that does not match the stream the `awslogs` driver writes (for example the
prefix without the container name):
```python
EcsRunTaskOperator(
task_id="repro",
cluster="my-cluster",
task_definition="my-task-def", # awslogs driver, stream prefix "ecs",
container "app"
launch_type="FARGATE",
overrides={"containerOverrides": [{"name": "app", "command": ["sh",
"-c", "exit 3"]}]},
network_configuration={...},
awslogs_group="/my/log-group",
awslogs_stream_prefix="ecs/wrong", # the stream is ecs/app/<task id>
skip_on_exit_code=[3],
)
```
Result:
```
botocore.errorfactory.ResourceNotFoundException: An error occurred
(ResourceNotFoundException) when calling the GetLogEvents operation: The
specified log stream does not exist.
```
With the right prefix the same task raises `AirflowSkipException` as
expected (`This task is not in success state - last 10 logs from Cloudwatch:`).
A container that cannot start at all is not affected: it raises
`EcsTaskFailToStart` before the log lines are read.
### What you think should happen instead?
A missing log stream should not hide the task's outcome:
`get_last_log_messages` should treat it as no log lines, as the polling loop
does, so the operator raises its usual exception, or `AirflowSkipException` for
an exit code in `skip_on_exit_code`, possibly with a warning that the stream
was not found.
### Operating System
macOS (Airflow run locally against ECS Fargate in eu-west-1)
### Deployment
Other
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.37.0
### Anything else?
`BatchOperator` calls the same method inside a `try` that logs a warning
("Unable to retrieve CloudWatch logs for AWS Batch job"), so it is not
affected; the ECS failure path could do the same.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
