camilleanne opened a new issue, #25522: URL: https://github.com/apache/airflow/issues/25522
### Description Support [multinode job types](https://docs.aws.amazon.com/batch/latest/userguide/multi-node-parallel-jobs.html) in the [AWS Batch Operator](https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/operators/batch.py). The [boto3 `submit_job` method](https://boto3.amazonaws.com/v1/documentation/api/1.9.88/reference/services/batch.html#Batch.Client.submit_job) supports container, multinode, and array batch jobs with the mutually exclusive `nodeOverrides` and `containerOverrides` (+ `arrayProperties`) parameters. But currently the AWS Batch Operator only supports submission of container jobs and array jobs by hardcoding the boto3 `submit_job` parameter `containerOverrides`: https://github.com/apache/airflow/blob/3c08cefdfd2e2636a714bb835902f0cb34225563/airflow/providers/amazon/aws/operators/batch.py#L200 & https://github.com/apache/airflow/blob/3c08cefdfd2e2636a714bb835902f0cb34225563/airflow/providers/amazon/aws/hooks/batch_client.py#L99 The [`get_job_awslogs_info`](https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/hooks/batch_client.py#L419) method in the batch client hook is also hardcoded for the container type job: https://github.com/apache/airflow/blob/3c08cefdfd2e2636a714bb835902f0cb34225563/airflow/providers/amazon/aws/hooks/batch_client.py#L425 To support multinode jobs the `get_job_awslogs_info` method would need to access `nodeProperties` from the [`describe_jobs`](https://boto3.amazonaws.com/v1/documentation/api/1.9.88/reference/services/batch.html#Batch.Client.describe_jobs) response. ### Use case/motivation Multinode jobs are a supported job type of AWS Batch, are supported by the underlying boto3 library, and should be also be available to be managed by Airflow. I've extended the AWS Batch Operator for our own use cases, but would prefer to not maintain a separate operator. ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
