dstandish commented on a change in pull request #14027:
URL: https://github.com/apache/airflow/pull/14027#discussion_r570684518
##########
File path: airflow/providers/amazon/aws/hooks/glue.py
##########
@@ -60,9 +62,10 @@ def __init__(
num_of_dpus: int = 10,
region_name: Optional[str] = None,
iam_role_name: Optional[str] = None,
+ connections: List[str] = None,
Review comment:
i think it might make sense to instead add `job_kwargs` because there
are [other kwargs that users might want to
specify](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.create_job)
a side benefit of doing so is we avoid the confusion of airflow connection
vs glue connection
i would also suggest that you might consider making the job kwargs
configurable as extra params in the airflow connection object so that something
like a subnet could be specified in the airflow connection and not hardcoded.
this would also mean that the operator would not necessarily need the
`job_kwargs` param though including it there also does not hurt
##########
File path: airflow/providers/amazon/aws/hooks/glue.py
##########
@@ -60,9 +62,10 @@ def __init__(
num_of_dpus: int = 10,
region_name: Optional[str] = None,
iam_role_name: Optional[str] = None,
+ connections: List[str] = None,
Review comment:
i think it might make sense to instead add `job_kwargs` because there
are [other kwargs that users might want to
specify](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.create_job)
a side benefit of doing so is we avoid the confusion of airflow connection
vs glue connection
i would also suggest that you might consider making the job kwargs
configurable as extra params in the airflow connection object so that something
like a subnet could be specified in the airflow connection and not only
configurable through python. this would also mean that the operator would not
necessarily need the `job_kwargs` param though including it there also does not
hurt
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
