kumbarsg opened a new issue, #25996:
URL: https://github.com/apache/airflow/issues/25996
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
I tried with all the versions of airflow providers
### Apache Airflow version
2.2.2
### Operating System
on AWS
### Deployment
MWAA
### Deployment details
_No response_
### What happened
I am using Airflow Amazon provider packages on my MWAA with airflow version
2.2.2 and Aws providers version 3.2.0(also tried with further versions)
I am trying to create a glue job with GlueJobOperator with this
configuration 'NumberOfWorkers': 2, 'WorkerType': 'G.1X'. Here's my code for
job creation.
```
job_name = "glue_job"
submit_glue_job = AwsGlueJobOperator(
task_id="glue_job",
job_name=job_name,
wait_for_completion=True,
# num_of_dpus=10,
retry_limit=0,
script_location=f"s3://bucket/etl.py",
s3_bucket=GLUE_EXAMPLE_S3_BUCKET,
iam_role_name=GLUE_CRAWLER_ROLE.split("/")[-1],
create_job_kwargs={
'GlueVersion': '3.0', 'NumberOfWorkers': 2, 'WorkerType': 'G.1X',
"DefaultArguments": {"--enable-glue-datacatalog": ''}
}
```
and here's the error:
```
when calling the CreateJob operation:
Please do not set Allocated Capacity if using Worker Type and Number of
Workers
```
I checked the official documentation to see if the Allocated capacity is
assigned to any default value, but it's not.
- Also, it is not updating an existing job's parameters
### What you think should happen instead
It should create a glue job with provided parameters but an error is showing
up saying I cannot assign values to number of workers and worker type along
with Allocated capacity, which I am not assigning in my code.
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
