Taragolis commented on PR #28816: URL: https://github.com/apache/airflow/pull/28816#issuecomment-1382212054
> I didn't find other examples of this pattern in the AWS package, RDS, Sagemaker and S3 are the only 3 that popped up, but also it's a bit hard to grep for this. I know a lot of samples where same approaches uses... but all of them inside of huge object structure. For example in Batch client: [SubmitJob](https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html) 1. Environment variables in [containerOverrides](https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html#Batch-SubmitJob-request-containerOverrides), [EksContainerOverride](https://docs.aws.amazon.com/batch/latest/APIReference/API_EksContainerOverride.html#Batch-Type-EksContainerOverride-env) expected list of dict with "name" and "value" keys 2. [resourceRequirements](https://docs.aws.amazon.com/batch/latest/APIReference/API_ContainerOverrides.html#Batch-Type-ContainerOverrides-resourceRequirements) expected list of dict with "type" and "value" keys [RegisterJobDefinition](https://docs.aws.amazon.com/batch/latest/APIReference/API_RegisterJobDefinition.html) 1. [environment](https://docs.aws.amazon.com/batch/latest/APIReference/API_ContainerProperties.html#Batch-Type-ContainerProperties-environment) expected list of dict with "name" and "value" keys 2. [secrets](https://docs.aws.amazon.com/batch/latest/APIReference/API_ContainerProperties.html#Batch-Type-ContainerProperties-secrets) expected list of dict with "name" and "valueFrom" keys So end users might also use this function for produce this objects (I don't think we should help them with this) by just provide dict and appropriate mapping for key_label and value_label. https://github.com/apache/airflow/blob/e6a037956cb31a883fddc2f9ae9ad5fc0ce541bf/airflow/providers/amazon/aws/utils/aws_api.py#L22 Some almost same function I use in the past in Airflow 1.10.x: ```python def aws_batch_env(d): return [{"name": k, "value": v} for k, v in d.items()] ... batch_env = { "VAR_1": "{{ ti.xcom_pull(task_ids='foo') }}", "VAR_2": "{{ ti.xcom_pull(task_ids='bar') }}", ... "VAR_n": "{{ ti.xcom_pull(task_ids='spam') }}", } batch_task = BatchOperator( ... overrides={ "environment": aws_batch_env(batch_env) }, ) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
