andrew-parsons-janus opened a new issue, #35637:
URL: https://github.com/apache/airflow/issues/35637
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
# Versions
Airflow version: 2.4.3
apache-airflow-providers-amazon==8.9.0
# Airflow error
```
Broken DAG: [/opt/airflow/dags/<project_directory>/<dag>.py] Traceback (most
recent call last):
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/models/baseoperator.py",
line 408, in apply_defaults
result = func(self, **kwargs, default_args=default_args)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/models/baseoperator.py",
line 756, in __init__
raise AirflowException(
airflow.exceptions.AirflowException: Invalid arguments were passed to
GlueJobOperator (task_id: submit_glue_job). Invalid arguments were:
**kwargs: {'update_config': True}
```
### What you think should happen instead
I don't believe passing an argument to `GlueJobOperator`'s `update_config`
init parameter should throw an exception. I also don't believe that I should
have to set `ALLOW_ILLEGAL_ARGUMENTS`. Why would a documented parameter be
illegal?
If this is expected behavior, then it should be better documented.
### How to reproduce
## Airflow task
```python3
@task_group(group_id="...", default_args=None)
def run_glue_job(job_name: str, script_args: dict):
submit_glue_job = GlueJobOperator(
# BaseOperator kwargs
task_id="submit_glue_job",
retries=0,
wait_for_completion=False,
on_success_callback=foo,
on_failure_callback=[foo, bar],
# GlueJobOperator kwargs
job_name=job_name,
script_location=f"{S3_PATH/glue_script.py",
script_args=script_args,
update_config=True, # <-- this causes an exception!
**EXTRA_KWARGS_GLUE,
)
wait_on_glue_job = GlueJobSensor(
task_id="wait_on_glue_job",
job_name=job_name,
run_id=submit_glue_job.output, # type: ignore
on_failure_callback=bar,
)
submit_glue_job >> wait_on_glue_job
```
### Operating System
macOS 14.1.1
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.9.0
### Deployment
Docker-Compose
### Deployment details
Docker version 20.10.24, build 297e128
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
