Taragolis commented on PR #29522: URL: https://github.com/apache/airflow/pull/29522#issuecomment-1430401332
There is 3 different sets of parameters for [SubmitJob](https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html) 1. [containerOverrides](https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html#API_SubmitJob_RequestSyntax) which run batch job on either EC2 or Fargate 2. [nodeOverrides](https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html#API_SubmitJob_RequestSyntax) which run batch job in EC2. This property include Has own `containerOverrides` 3. [eksPropertiesOverride](https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html#API_SubmitJob_RequestSyntax) run batch job on EKS cluster I also guess that [arrayProperties](https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html#API_SubmitJob_RequestSyntax) only applicable for `containerOverrides` IMHO SubmitJob it is pretty complicated. One potential benefit for keep all in one operator it is ability to set upstream task create kwargs for `BatchOperator.partial(...).expand_kwargs(...)`. But right now BatchOperator can't work with Dynamic Tasks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
