SamWheating edited a comment on issue #20832: URL: https://github.com/apache/airflow/issues/20832#issuecomment-1011502821
I think that this is because the GlueHook is pretty opinionated and hardcodes the value of `Command` when running the `glue_client.create_job` command: https://github.com/apache/airflow/blob/2ab2ae8849bf6d80a700b1b74cef37eb187161ad/airflow/providers/amazon/aws/hooks/glue.py#L181-L225 So when you provide `Command` as a `create_job_kwargs`, it ends up being supplied twice to that function (Although I suspect that this would be a typeError, not a keyError 🤔) Anyways, thoughts on just making the `Command.Name` and `Command.PythonVersion` argument configurable in the `GlueJobOperator`? If y'all think that this is a satisfactory fix, feel free to assign this issue to me and I can put up a quick PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
