o-nikolas commented on issue #35490:
URL: https://github.com/apache/airflow/issues/35490#issuecomment-1804920164
I started working on this one and there is a bit of nuance that I didn't
expect.
Many of the config options are lists of dicts. Take tags for example, the
type is:
```python
tags=[
{
'key': 'string',
'value': 'string'
},
],
```
So if you have a base set of run_task_kwargs configured in your airflow
config such as:
```shell
AIRFLOW__AWS_ECS_EXECUTOR__RUN_TASK_KWARGS='{
"tags": [{"key": "fish", "value": "banana"}, {"key": "peanut", "value":
"sauce"}],
"startedBy": "Niko",
"overrides": {"containerOverrides": [{"memory": 500, "environment":[{"name":
"FOO", "value": "BAR"}]}]}}'
```
And in your `executor_config` on a particular task you specify:
```python
@task(
executor_config={
"tags": [{"key": "FOO", "value": "BAR"}],
"overrides": {"containerOverrides": [{"memory": 740}]},
}
)
def hello_world():
print("hello_world")
```
Should the final kwargs sent to ECS for **tags** be the sum of the two?:
`tags=[{'key': 'fish', 'value': 'banana'}, {'key': 'peanut', 'value': 'sauce'},
{"key": "FOO", "value": "BAR"}]` or a zipping of the two where we update each
dict in the list by index (copying/leaving the remainder if one list is smaller
than the other): `tags=[{'key': 'FOO', 'value': 'BAR'}, {'key': 'peanut',
'value': 'sauce'}]` or should the `executor_config` completely override from?:
`"tags": [{'key': 'FOO', 'value': 'BAR'}],`
There are many other configs that are lists of dicts in the ECS run_task
API, I'm not sure one answer fits all of them. containerOverrides for example
cannot be additive, we must update each dict with what is provided in the
`executor_config` since each index in the list maps to a container, so a
zipping update to each corresponding dict by index makes the most sense.
However, something like tags above seems like it could be a different approach.
We must evaluate many of the other configs as well (the full list is
[here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs/client/run_task.html)).
I'd very much appreciate some input from @ferruzzi @shubham22 @syedahsn
As well as from @mshober who has an implementation of this, so I'd be
interested to hear what approach they took and how it's working out for them:
> My version of the ECS Executor allows users to set the appropriate
Capacity Provider via the operator's executor_config param
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
