aischeveva opened a new issue, #32404:
URL: https://github.com/apache/airflow/issues/32404
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
### Airflow version
2.4.3, 2.5.0
### GlueJobOperator fails to create a job
With provider version 7.1.0 and above (I have also tried 7.4.1 and 8.2.0),
GlueJobOperator doesn't recognize the type of the GlueVersion in
create_job_kwargs correctly and fails to create a job. The passed parameter is
string, but validation fails claiming that it's a float.
```
[2023-07-06, 10:52:27 UTC] {{taskinstance.py:1851}} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/operators/glue.py",
line 146, in execute
glue_job_run = glue_job.initialize_job(self.script_args,
self.run_job_kwargs)
File
"/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/glue.py",
line 155, in initialize_job
job_name = self.create_or_update_glue_job()
File
"/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/glue.py",
line 303, in create_or_update_glue_job
self.update_job(**config)
File
"/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/glue.py",
line 289, in update_job
self.get_conn().update_job(JobName=job_name, JobUpdate=job_kwargs)
File
"/usr/local/airflow/.local/lib/python3.10/site-packages/botocore/client.py",
line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File
"/usr/local/airflow/.local/lib/python3.10/site-packages/botocore/client.py",
line 919, in _make_api_call
request_dict = self._convert_to_request_dict(
File
"/usr/local/airflow/.local/lib/python3.10/site-packages/botocore/client.py",
line 990, in _convert_to_request_dict
request_dict = self._serializer.serialize_to_request(
File
"/usr/local/airflow/.local/lib/python3.10/site-packages/botocore/validate.py",
line 381, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter JobUpdate.GlueVersion, value: 3.0, type: <class
'float'>, valid types: <class 'str'>
```
I have observed this problem both in local setup with virtual environment on
WSL Ubuntu 20.04 LTS with Airflow 2.5.0 and in Amazon MWAA with Airflow 2.4.3.
### What you think should happen instead
I would expect it shouldn't convert GlueVersion string to float when
creating job to then fail its validation. The same code works fine with
apache-airflow-providers-amazon==6.0.0, so I would expect same behavior.
### How to reproduce
Create a DAG and add a
```
GlueJobOperator(
task_id='submit_glue_job',
job_name='job-name',
script_location='script-location',
s3_bucket='your-s3-bucket',
iam_role_name='your-glue-role-name',
create_job_kwargs={"GlueVersion": "3.0"}
)
```
### Operating System
WSL Ubuntu 20.04 LTS / Linux
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==7.1.0
### Deployment
Amazon (AWS) MWAA
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
