TamGB opened a new issue, #24904:
URL: https://github.com/apache/airflow/issues/24904
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
4.00
### Apache Airflow version
2.3.2 (latest released)
### Operating System
Ubuntu
### Deployment
Docker-Compose
### Deployment details
_No response_
### What happened
Using the `EcsOperator` fails with misconfiguration errors even though the
AWS connection is setup as such:
```
[2022-07-07, 08:00:02 UTC] {ecs.py:294} INFO - Running ECS Task - Task
definition: XXXX - on cluster YYYY
[2022-07-07, 08:00:02 UTC] {ecs.py:296} INFO - EcsOperator overrides:
{'containerOverrides': [{'name': 'XXXX', 'command': ['mycommands'],
'environment': [{'AWS_ACCESS_KEY_ID': 'XXXX', 'AWS_SECRET_ACCESS_KEY': 'ZZZZ',
'AWS_REGION': 'us-west-2'}]}]}
[2022-07-07, 08:00:02 UTC] {base.py:68} INFO - Using connection ID
'aws_ecs_conn' for task execution.
[2022-07-07, 08:00:02 UTC] {base_aws.py:206} INFO - Credentials retrieved
from login
[2022-07-07, 08:00:02 UTC] {base_aws.py:100} INFO - Retrieving region_name
from Connection.extra_config['region_name']
[2022-07-07, 08:00:02 UTC] {taskinstance.py:1889} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py",
line 71, in wrapper
return func(*args, session=session, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/ecs.py",
line 303, in execute
self._start_wait_check_task(context)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py",
line 582, in decorator_f
return fun(self, *args, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/ecs.py",
line 321, in _start_wait_check_task
self._start_task(context)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/ecs.py",
line 369, in _start_task
response = self.client.run_task(**run_opts)
File
"/home/airflow/.local/lib/python3.7/site-packages/botocore/client.py", line
508, in _api_call
return self._make_api_call(operation_name, kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/botocore/client.py", line
875, in _make_api_call
api_params, operation_model, context=request_context
File
"/home/airflow/.local/lib/python3.7/site-packages/botocore/client.py", line
936, in _convert_to_request_dict
api_params, operation_model
File
"/home/airflow/.local/lib/python3.7/site-packages/botocore/validate.py", line
381, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in overrides.containerOverrides[0].environment[0]:
"AWS_ACCESS_KEY_ID", must be one of: name, value
Unknown parameter in overrides.containerOverrides[0].environment[0]:
"AWS_SECRET_ACCESS_KEY", must be one of: name, value
Unknown parameter in overrides.containerOverrides[0].environment[0]:
"AWS_REGION", must be one of: name, value
[2022-07-07, 08:00:02 UTC] {taskinstance.py:1400} INFO - Marking task as
UP_FOR_RETRY. dag_id=ecs_dag, task_id=a_task, execution_date=20220706T080000,
start_date=20220707T080001, end_date=20220707T080002
```
### What you think should happen instead
The task should not fail.
### How to reproduce
Setup an `Amazon Web Service` connection and setup an ECSOperator Task:
```
ope = EcsOperator(
task_id=f'docker_{test}',
cluster=cluster_name,
task_definition=task_definition_name,
launch_type="EXTERNAL|FARGATE",
aws_conn_id="aws_ecs",
overrides={
"containerOverrides": [
{
"name": "XXXX,
"command": ["mycommand"],
"environment": [
{
'AWS_ACCESS_KEY_ID': *****,
'AWS_SECRET_ACCESS_KEY': ****,
'AWS_REGION': *****,
},
],
},
],
},
tags={
"Project": "Test",
},
awslogs_group="/ecs/hello-world",
awslogs_region=s3_logs_region,
awslogs_stream_prefix="ecs/hello-world-container"
)
```
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
