[
https://issues.apache.org/jira/browse/AIRFLOW-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657234#comment-16657234
]
ASF GitHub Bot commented on AIRFLOW-1788:
-----------------------------------------
ashb closed pull request #2760: [AIRFLOW-1788] Support all boto kwargs
URL: https://github.com/apache/incubator-airflow/pull/2760
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/hooks/emr_hook.py
b/airflow/contrib/hooks/emr_hook.py
index cee239867c..2acb33451e 100644
--- a/airflow/contrib/hooks/emr_hook.py
+++ b/airflow/contrib/hooks/emr_hook.py
@@ -56,7 +56,8 @@ def create_job_flow(self, job_flow_overrides):
VisibleToAllUsers=config.get('VisibleToAllUsers'),
JobFlowRole=config.get('JobFlowRole'),
ServiceRole=config.get('ServiceRole'),
- Tags=config.get('Tags')
+ Tags=config.get('Tags'),
+ SecurityConfiguration=config.get('SecurityConfiguration'),
)
return response
diff --git a/tests/contrib/hooks/test_emr_hook.py
b/tests/contrib/hooks/test_emr_hook.py
index 119df99a7c..ab1387e52a 100644
--- a/tests/contrib/hooks/test_emr_hook.py
+++ b/tests/contrib/hooks/test_emr_hook.py
@@ -48,6 +48,19 @@ def
test_create_job_flow_uses_the_emr_config_to_create_a_cluster(self):
cluster = hook.create_job_flow({'Name': 'test_cluster'})
self.assertEqual(client.list_clusters()['Clusters'][0]['Id'],
cluster['JobFlowId'])
+
+ @unittest.skipIf(mock_emr is None, 'mock_emr package not present')
+ @mock_emr
+ def test_create_job_flow_passes_through_boto_kwargs(self):
+ client = boto3.client('emr', region_name='us-east-1')
+ if len(client.list_clusters()['Clusters']):
+ raise ValueError('AWS not properly mocked')
+
+ hook = EmrHook(aws_conn_id='aws_default', emr_conn_id='emr_default')
+ cluster = hook.create_job_flow({'Name': 'test_cluster',
+ 'SecurityConfiguration':
'test_config'})
+
+ self.assertEqual(client.list_clusters()['Clusters'][0]['Id'],
cluster['JobFlowId'])
if __name__ == '__main__':
unittest.main()
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> EMR hook doesn't pass through SecurityConfiguration parameter
> -------------------------------------------------------------
>
> Key: AIRFLOW-1788
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1788
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Sam Kitajima-Kimbrel
> Assignee: Sam Kitajima-Kimbrel
> Priority: Major
>
> Expected: `SecurityConfiguration` argument to boto3's run_job_flow() method
> are passed through via the EMR hook in Airflow.
> Actual: Not passed through because hard-coded parameter names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
