5e9148d9 opened a new issue, #37278:
URL: https://github.com/apache/airflow/issues/37278
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
8.7.1
### Apache Airflow version
2.7.2
### Operating System
AWS service
### Deployment
Amazon (AWS) MWAA
### Deployment details
_No response_
### What happened
Python DAG file defines EcsRunTaskOperator.awslogs_group="my-loggroup". And
we got error in logs
`
...
[2024-02-08 18:56:36,345] Cannot find log stream yet, it can take a couple
of seconds to show up. If this error persists, check that the log group and
stream are correct: group: my-loggroup stream:
ecs/my-container/b478044071514a32b3bf8c7dee5ebea4
[2024-02-08 18:56:36,420] ECS Task stopped, check status: {removed}
[2024-02-08 18:56:36,485] Task failed with exception
Traceback (most recent call last):
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/operators/ecs.py",
line 585, in execute
self._after_execution()
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/operators/ecs.py",
line 610, in _after_execution
self._check_success_task()
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py",
line 746, in decorator_f
return fun(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/operators/ecs.py",
line 731, in _check_success_task
self.task_log_fetcher.get_last_log_messages(self.number_logs_exception)
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/utils/task_log_fetcher.py",
line 103, in get_last_log_messages
response = self.hook.conn.get_log_events(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/botocore/client.py",
line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/botocore/client.py",
line 980, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ResourceNotFoundException: An error occurred
(ResourceNotFoundException) when calling the GetLogEvents operation: The
specified log stream does not exist.
[2024-02-08 18:56:36,515] Marking task as UP_FOR_RETRY.
dag_id=rewards_rec_pipeline_DEV, task_id=data_update,
execution_date=20240208T185300, start_date=20240208T185304,
end_date=20240208T185636
[2024-02-08 18:56:36,540] Failed to execute job 111 for task data_update (An
error occurred (ResourceNotFoundException) when calling the GetLogEvents
operation: The specified log stream does not exist.; 172)
[2024-02-08 18:56:36,649] Task exited with return code 1
[2024-02-08 18:56:36,689] 0 downstream tasks scheduled from follow-on
schedule check
`
Issue that fargate task is created with another log group and value from
EcsRunTaskOperator.awslogs_group does not override it
### What you think should happen instead
Value EcsRunTaskOperator.awslogs_group should override default fargate task
definition log group name.
### How to reproduce
1) Create python DAG which run ecs fargate task.
2) Set custom log group name in EcsRunTaskOperator.awslogs_group (should be
different from log group name defined in fargate task definition)
3) Run the pipeline and get traceback
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
