Ritika-Singhal commented on pull request #19787: URL: https://github.com/apache/airflow/pull/19787#issuecomment-977161677
> Instead of sliently dropping `num_of_dpus`, I feel we should explicitly fail with `ValueError` in `__init__`. In the `AwsGlueJobHook` `__init__`, the `num_of_dpus` are being initialized by default to 10 if the user did not specify this parameter. So, to throw a `ValueError`, we would need to remove the default initialization of `num_of_dpus` (otherwise it will always throw the error because `num_of_dpus` will always have a valid value). It will also need to check if the user has specified one of the following arguments or not i.e. either (`num_of_dpus`) or (`WorkerType` and `NumberOfWorkers`) but not both. The default initialization of the `num_of_dpus` is keeping the job from failing if the user did not specify either of the parameters. Currently, the `num_of_dpus` parameter works as a default option in the `else` block if the user did not specify the `WorkerType` and `NumberOfWorkers`. So, it's not completed dropped. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
