Ritika-Singhal commented on issue #19207: URL: https://github.com/apache/airflow/issues/19207#issuecomment-951566516
Thanks for acknowledging this issue. The solution which I have in mind to correct this is: In the AWSGlueJobHook script where `glue_client.create_job` is being called, we need to insert an if else block which checks whether the `create_job_kwargs` dict has the `WorkerType` and `NumberOfWorkers` parameters. If it has these 2 parameters, then in the `glue_client.create_job` function call, we need to omit the `AllocatedCapacity` parameter. However, if these 2 parameters are not there win the `create_job_kwargs` dict , then we should have the function call assigning the `num_of_dpus` to either `AllocatedCapacity` or `MaxCapacity` parameter. So, my plan includes modifying function call based on the if else condition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
