[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r371262602
##
File path: airflow/providers/amazon/aws/hooks/cloud_formation.py
##
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains AWS CloudFormation Hook
+"""
+from botocore.exceptions import ClientError
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSCloudFormationHook(AwsHook):
+"""
+Interact with AWS CloudFormation.
+"""
+
+def __init__(self, region_name=None, *args, **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+if not self.conn:
+self.conn = self.get_client_type('cloudformation',
self.region_name)
+return self.conn
+
+def get_stack_status(self, stack_name):
+"""
+Get stack status from CloudFormation.
+"""
+cloudformation = self.get_conn()
+
+self.log.info('Poking for stack %s', stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=stack_name)['Stacks']
+return stacks[0]['StackStatus']
+except ClientError as e:
+if 'does not exist' in str(e):
+return None
+else:
+raise e
Review comment:
We could check the error code instead of the message string, and assume that
`ValidationError` returned from `describe_stacks` could only be due to stack
not existing.
However there could be other scenarios in which that can happen, for
example, illegal characters in `StackName`:
```
>>> cf.describe_stacks(StackName='||')
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
661, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationError) when
calling the DescribeStacks operation: 1 validation error detected: Value '||'
at 'stackName' failed to satisfy constraint: Member must satisfy regular
expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r371262602
##
File path: airflow/providers/amazon/aws/hooks/cloud_formation.py
##
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains AWS CloudFormation Hook
+"""
+from botocore.exceptions import ClientError
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSCloudFormationHook(AwsHook):
+"""
+Interact with AWS CloudFormation.
+"""
+
+def __init__(self, region_name=None, *args, **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+if not self.conn:
+self.conn = self.get_client_type('cloudformation',
self.region_name)
+return self.conn
+
+def get_stack_status(self, stack_name):
+"""
+Get stack status from CloudFormation.
+"""
+cloudformation = self.get_conn()
+
+self.log.info('Poking for stack %s', stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=stack_name)['Stacks']
+return stacks[0]['StackStatus']
+except ClientError as e:
+if 'does not exist' in str(e):
+return None
+else:
+raise e
Review comment:
We could check the error code instead of the message string, and assume that
`ValidationError` returned from `describe_stacks` could only be due to stack
not existing.
However there could be other scenarios in which that can happen, for
example, illegal characters in `StackName`:
```
>>> cf.describe_stacks(StackName='||')
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
661, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationError) when
calling the DescribeStacks operation: 1 validation error detected: Value '||'
at 'stackName' failed to satisfy constraint: Member must satisfy regular
expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r371262602
##
File path: airflow/providers/amazon/aws/hooks/cloud_formation.py
##
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains AWS CloudFormation Hook
+"""
+from botocore.exceptions import ClientError
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSCloudFormationHook(AwsHook):
+"""
+Interact with AWS CloudFormation.
+"""
+
+def __init__(self, region_name=None, *args, **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+if not self.conn:
+self.conn = self.get_client_type('cloudformation',
self.region_name)
+return self.conn
+
+def get_stack_status(self, stack_name):
+"""
+Get stack status from CloudFormation.
+"""
+cloudformation = self.get_conn()
+
+self.log.info('Poking for stack %s', stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=stack_name)['Stacks']
+return stacks[0]['StackStatus']
+except ClientError as e:
+if 'does not exist' in str(e):
+return None
+else:
+raise e
Review comment:
We could check the error code instead of the message string, and assume that
`ValidationError` returned from `describe_stacks` could only be due to stack
not existing.
However potentially there be other scenarios in which that can happen, for
example, illegal characters in `StackName`:
```
>>> cf.describe_stacks(StackName='||')
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
661, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationError) when
calling the DescribeStacks operation: 1 validation error detected: Value '||'
at 'stackName' failed to satisfy constraint: Member must satisfy regular
expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r371262602
##
File path: airflow/providers/amazon/aws/hooks/cloud_formation.py
##
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains AWS CloudFormation Hook
+"""
+from botocore.exceptions import ClientError
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSCloudFormationHook(AwsHook):
+"""
+Interact with AWS CloudFormation.
+"""
+
+def __init__(self, region_name=None, *args, **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+if not self.conn:
+self.conn = self.get_client_type('cloudformation',
self.region_name)
+return self.conn
+
+def get_stack_status(self, stack_name):
+"""
+Get stack status from CloudFormation.
+"""
+cloudformation = self.get_conn()
+
+self.log.info('Poking for stack %s', stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=stack_name)['Stacks']
+return stacks[0]['StackStatus']
+except ClientError as e:
+if 'does not exist' in str(e):
+return None
+else:
+raise e
Review comment:
We could check the error code instead of the message string, and assume that
`ValidationError` returned from `describe_stacks` could only be due to stack
not existing.
However potentially there be other scenarios in which that can happen, for
example, illegal characters in `StackName`:
```>>> cf.describe_stacks(StackName='||')
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
661, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationError) when
calling the DescribeStacks operation: 1 validation error detected: Value '||'
at 'stackName' failed to satisfy constraint: Member must satisfy regular
expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r371262602
##
File path: airflow/providers/amazon/aws/hooks/cloud_formation.py
##
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains AWS CloudFormation Hook
+"""
+from botocore.exceptions import ClientError
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSCloudFormationHook(AwsHook):
+"""
+Interact with AWS CloudFormation.
+"""
+
+def __init__(self, region_name=None, *args, **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+if not self.conn:
+self.conn = self.get_client_type('cloudformation',
self.region_name)
+return self.conn
+
+def get_stack_status(self, stack_name):
+"""
+Get stack status from CloudFormation.
+"""
+cloudformation = self.get_conn()
+
+self.log.info('Poking for stack %s', stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=stack_name)['Stacks']
+return stacks[0]['StackStatus']
+except ClientError as e:
+if 'does not exist' in str(e):
+return None
+else:
+raise e
Review comment:
We could check the error code instead of the message string, and assume that
`ValidationError` returned from `describe_stacks` could only be due to stack
not existing.
However potentially there be other scenarios in which that can happen, for
example, illegal characters in `StackName`:
```>>> cf.describe_stacks(StackName='||')
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line
661, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationError) when
calling the DescribeStacks operation: 1 validation error detected: Value '||'
at 'stackName' failed to satisfy constraint: Member must satisfy regular
expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369977013
##
File path: airflow/providers/amazon/aws/hooks/cloud_formation.py
##
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains AWS CloudFormation Hook
+"""
+from botocore.exceptions import ClientError
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSCloudFormationHook(AwsHook):
+"""
+Interact with AWS CloudFormation.
+"""
+
+def __init__(self, region_name=None, *args, **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+if not self.conn:
+self.conn = self.get_client_type('cloudformation',
self.region_name)
+return self.conn
+
+def get_stack_status(self, stack_name):
+"""
+Get stack status from CloudFormation.
+"""
+cloudformation = self.get_conn()
+
+self.log.info('Poking for stack %s', stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=stack_name)['Stacks']
+return stacks[0]['StackStatus']
+except ClientError as e:
+if 'does not exist' in str(e):
+return None
+else:
+raise e
Review comment:
Yep, I read that, however there is no class by name
`AmazonCloudFormationException` in boto.
I could add:
```
if e.response['Error']['Code'] == 'ValidationError':
```
as suggested [here](https://stackoverflow.com/a/47040476/2489287)
However since we are using boto's cloudformation client here it is certain
that that would be the value of the error code in the exception.
Should I make this change? or should we go for the brevity of simply
catching `ClientError` and not check its error code value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r371262602
##
File path: airflow/providers/amazon/aws/hooks/cloud_formation.py
##
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains AWS CloudFormation Hook
+"""
+from botocore.exceptions import ClientError
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSCloudFormationHook(AwsHook):
+"""
+Interact with AWS CloudFormation.
+"""
+
+def __init__(self, region_name=None, *args, **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+if not self.conn:
+self.conn = self.get_client_type('cloudformation',
self.region_name)
+return self.conn
+
+def get_stack_status(self, stack_name):
+"""
+Get stack status from CloudFormation.
+"""
+cloudformation = self.get_conn()
+
+self.log.info('Poking for stack %s', stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=stack_name)['Stacks']
+return stacks[0]['StackStatus']
+except ClientError as e:
+if 'does not exist' in str(e):
+return None
+else:
+raise e
Review comment:
We could check the error code instead of the message string, and assume that
`ValidationError` returned from `describe_stacks` could only be due to stack
not existing, however potentially there could be other scenarios in which that
can happen.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369977013
##
File path: airflow/providers/amazon/aws/hooks/cloud_formation.py
##
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains AWS CloudFormation Hook
+"""
+from botocore.exceptions import ClientError
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSCloudFormationHook(AwsHook):
+"""
+Interact with AWS CloudFormation.
+"""
+
+def __init__(self, region_name=None, *args, **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+if not self.conn:
+self.conn = self.get_client_type('cloudformation',
self.region_name)
+return self.conn
+
+def get_stack_status(self, stack_name):
+"""
+Get stack status from CloudFormation.
+"""
+cloudformation = self.get_conn()
+
+self.log.info('Poking for stack %s', stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=stack_name)['Stacks']
+return stacks[0]['StackStatus']
+except ClientError as e:
+if 'does not exist' in str(e):
+return None
+else:
+raise e
Review comment:
Yep, I read that, however there is no class by name
`AmazonCloudFormationException` in boto.
I could add:
```
if e.response['Error']['Code'] == 'AmazonCloudFormationException':
```
as suggested [here](https://stackoverflow.com/a/47040476/2489287)
However since we are using boto's cloudformation client here it is certain
that that would be the value of the error code in the exception.
Should I make this change? or should we go for the brevity of simply
catching `ClientError` and not check its error code value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369977013
##
File path: airflow/providers/amazon/aws/hooks/cloud_formation.py
##
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains AWS CloudFormation Hook
+"""
+from botocore.exceptions import ClientError
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSCloudFormationHook(AwsHook):
+"""
+Interact with AWS CloudFormation.
+"""
+
+def __init__(self, region_name=None, *args, **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+if not self.conn:
+self.conn = self.get_client_type('cloudformation',
self.region_name)
+return self.conn
+
+def get_stack_status(self, stack_name):
+"""
+Get stack status from CloudFormation.
+"""
+cloudformation = self.get_conn()
+
+self.log.info('Poking for stack %s', stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=stack_name)['Stacks']
+return stacks[0]['StackStatus']
+except ClientError as e:
+if 'does not exist' in str(e):
+return None
+else:
+raise e
Review comment:
Yep, I read that, however there is no class by name
AmazonCloudFormationException in boto.
I could add:
```
if e.response['Error']['Code'] == 'AmazonCloudFormationException':
```
as suggested [here](https://stackoverflow.com/a/47040476/2489287)
However since we are using boto's cloudformation client here it is certain
that that would be the value of the error code in the exception.
Should I make this change? or should we go for the brevity of simply
catching `ClientError` and not check its error code value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369977013
##
File path: airflow/providers/amazon/aws/hooks/cloud_formation.py
##
@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains AWS CloudFormation Hook
+"""
+from botocore.exceptions import ClientError
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSCloudFormationHook(AwsHook):
+"""
+Interact with AWS CloudFormation.
+"""
+
+def __init__(self, region_name=None, *args, **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+if not self.conn:
+self.conn = self.get_client_type('cloudformation',
self.region_name)
+return self.conn
+
+def get_stack_status(self, stack_name):
+"""
+Get stack status from CloudFormation.
+"""
+cloudformation = self.get_conn()
+
+self.log.info('Poking for stack %s', stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=stack_name)['Stacks']
+return stacks[0]['StackStatus']
+except ClientError as e:
+if 'does not exist' in str(e):
+return None
+else:
+raise e
Review comment:
Yep, I read that, however there is no class by name
AmazonCloudFormationException in boto.
I could add:
```
e.response['Error']['Code'] == 'AmazonCloudFormationException'
```
as suggested [here](https://stackoverflow.com/a/47040476/2489287)
However since we are using boto's cloudformation client here it is certain
that that would be the value of the error code in the exception.
Should I make this change? or should we go for the brevity of simply
catching `ClientError` and not check its error code value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369972454
##
File path: airflow/providers/amazon/aws/sensors/cloud_formation.py
##
@@ -0,0 +1,98 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains sensors for AWS CloudFormation.
+"""
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class CloudFormationCreateStackSensor(BaseSensorOperator):
+"""
+Waits for a stack to be created successfully on AWS CloudFormation.
+
+:param stack_name: The name of the stack to wait for (templated)
+:type stack_name: str
+:param aws_conn_id: ID of the Airflow connection where credentials and
extra configuration are
+stored
+:type aws_conn_id: str
+:param poke_interval: Time in seconds that the job should wait between
each try
+:type poke_interval: int
+"""
+
+template_fields = ['stack_name']
+ui_color = '#C5CAE9'
+
+@apply_defaults
+def __init__(self,
+ stack_name,
+ aws_conn_id='aws_default',
+ region_name=None,
+ *args,
+ **kwargs):
+super().__init__(*args, **kwargs)
+self.stack_name = stack_name
+self.hook = AWSCloudFormationHook(aws_conn_id=aws_conn_id,
region_name=region_name)
+
+def poke(self, context):
+stack_status = self.hook.get_stack_status(self.stack_name)
+if stack_status == 'CREATE_COMPLETE':
+return True
+elif stack_status in ('CREATE_IN_PROGRESS', None):
+return False
+else:
+raise ValueError(f'Stack {self.stack_name} in bad state:
{stack_status}')
+
+
+class CloudFormationDeleteStackSensor(BaseSensorOperator):
+"""
+Waits for a stack to be deleted successfully on AWS CloudFormation.
+
+:param stack_name: The name of the stack to wait for (templated)
+:type stack_name: str
+:param aws_conn_id: ID of the Airflow connection where credentials and
extra configuration are
+stored
+:type aws_conn_id: str
+:param poke_interval: Time in seconds that the job should wait between
each try
+:type poke_interval: int
+"""
+
+template_fields = ['stack_name']
+ui_color = '#C5CAE9'
+
+@apply_defaults
+def __init__(self,
+ stack_name,
+ aws_conn_id='aws_default',
+ region_name=None,
+ *args,
+ **kwargs):
+super().__init__(*args, **kwargs)
+self.stack_name = stack_name
+self.hook = AWSCloudFormationHook(aws_conn_id=aws_conn_id,
region_name=region_name)
+
+def poke(self, context):
+stack_status = self.hook.get_stack_status(self.stack_name)
+if stack_status in ('DELETE_COMPLETE', None):
+return True
+elif stack_status == 'DELETE_IN_PROGRESS':
+return False
+else:
+raise ValueError(f'Stack {self.stack_name} in bad state:
{stack_status}')
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369972407
##
File path: airflow/providers/amazon/aws/sensors/cloud_formation.py
##
@@ -0,0 +1,98 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains sensors for AWS CloudFormation.
+"""
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class CloudFormationCreateStackSensor(BaseSensorOperator):
+"""
+Waits for a stack to be created successfully on AWS CloudFormation.
+
+:param stack_name: The name of the stack to wait for (templated)
+:type stack_name: str
+:param aws_conn_id: ID of the Airflow connection where credentials and
extra configuration are
+stored
+:type aws_conn_id: str
+:param poke_interval: Time in seconds that the job should wait between
each try
+:type poke_interval: int
+"""
+
+template_fields = ['stack_name']
+ui_color = '#C5CAE9'
+
+@apply_defaults
+def __init__(self,
+ stack_name,
+ aws_conn_id='aws_default',
+ region_name=None,
+ *args,
+ **kwargs):
+super().__init__(*args, **kwargs)
+self.stack_name = stack_name
+self.hook = AWSCloudFormationHook(aws_conn_id=aws_conn_id,
region_name=region_name)
+
+def poke(self, context):
+stack_status = self.hook.get_stack_status(self.stack_name)
+if stack_status == 'CREATE_COMPLETE':
+return True
+elif stack_status in ('CREATE_IN_PROGRESS', None):
+return False
+else:
+raise ValueError(f'Stack {self.stack_name} in bad state:
{stack_status}')
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369972551
##
File path: airflow/providers/amazon/aws/operators/cloud_formation.py
##
@@ -0,0 +1,125 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains CloudFormation create/delete stack operators.
+"""
+from typing import List
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.utils.decorators import apply_defaults
+
+
+class BaseCloudFormationOperator(BaseOperator):
+"""
+Base operator for CloudFormation operations.
+
+:param params: parameters to be passed to CloudFormation.
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#1d472b'
+ui_fgcolor = '#FFF'
+
+@apply_defaults
+def __init__(
+self,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(*args, **kwargs)
+self.params = params
+self.aws_conn_id = aws_conn_id
+
+def execute(self, context):
+self.log.info('Parameters: %s', self.params)
+
+
self.cloudformation_op(AWSCloudFormationHook(aws_conn_id=self.aws_conn_id))
+
+def cloudformation_op(self, cloudformation):
+"""
+This is the main method to run CloudFormation operation.
+"""
+raise NotImplementedError()
+
+
+class CloudFormationCreateStackOperator(BaseCloudFormationOperator):
+"""
+An operator that creates a CloudFormation stack.
+
+:param stack_name: stack name.
+:type params: str
+:param params: parameters to be passed to CloudFormation. For possible
arguments see:
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.create_stack
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#6b9659'
+
+@apply_defaults
+def __init__(
+self,
+stack_name,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(params=params, aws_conn_id=aws_conn_id, *args,
**kwargs)
+self.stack_name = stack_name
+
+def cloudformation_op(self, cloudformation):
+cloudformation.create_stack(self.stack_name, self.params)
+
+
+class CloudFormationDeleteStackOperator(BaseCloudFormationOperator):
+"""
+An operator that deletes a CloudFormation stack.
+
+:param stack_name: stack name.
+:type params: str
+:param params: parameters to be passed to CloudFormation. For possible
arguments see:
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.delete_stack
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#1d472b'
+ui_fgcolor = '#FFF'
+
+@apply_defaults
+def __init__(
+self,
+stack_name,
+params=None,
+aws_conn_id='aws_default',
+*args, **kwargs):
+if params is None:
+params = {}
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369971519
##
File path: airflow/providers/amazon/aws/operators/cloud_formation.py
##
@@ -0,0 +1,115 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains CloudFormation create/delete stack operators.
+"""
+from typing import List
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.utils.decorators import apply_defaults
+
+
+class BaseCloudFormationOperator(BaseOperator):
+"""
+Base operator for CloudFormation operations.
+
+:param params: parameters to be passed to CloudFormation.
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#1d472b'
+ui_fgcolor = '#FFF'
+
+@apply_defaults
+def __init__(
+self,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(*args, **kwargs)
+self.params = params
+self.aws_conn_id = aws_conn_id
+
+def execute(self, context):
+self.log.info('Parameters: %s', self.params)
+
+
self.cloudformation_op(AWSCloudFormationHook(aws_conn_id=self.aws_conn_id).get_conn())
+
+def cloudformation_op(self, cloudformation):
+"""
+This is the main method to run CloudFormation operation.
+"""
+raise NotImplementedError()
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369971576
##
File path: airflow/providers/amazon/aws/operators/cloud_formation.py
##
@@ -0,0 +1,125 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains CloudFormation create/delete stack operators.
+"""
+from typing import List
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.utils.decorators import apply_defaults
+
+
+class BaseCloudFormationOperator(BaseOperator):
+"""
+Base operator for CloudFormation operations.
+
+:param params: parameters to be passed to CloudFormation.
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#1d472b'
+ui_fgcolor = '#FFF'
+
+@apply_defaults
+def __init__(
+self,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(*args, **kwargs)
+self.params = params
+self.aws_conn_id = aws_conn_id
+
+def execute(self, context):
+self.log.info('Parameters: %s', self.params)
+
+
self.cloudformation_op(AWSCloudFormationHook(aws_conn_id=self.aws_conn_id))
+
+def cloudformation_op(self, cloudformation):
+"""
+This is the main method to run CloudFormation operation.
+"""
+raise NotImplementedError()
+
+
+class CloudFormationCreateStackOperator(BaseCloudFormationOperator):
+"""
+An operator that creates a CloudFormation stack.
+
+:param stack_name: stack name.
+:type params: str
+:param params: parameters to be passed to CloudFormation. For possible
arguments see:
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.create_stack
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#6b9659'
+
+@apply_defaults
+def __init__(
+self,
+stack_name,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(params=params, aws_conn_id=aws_conn_id, *args,
**kwargs)
+self.stack_name = stack_name
+
+def cloudformation_op(self, cloudformation):
+cloudformation.create_stack(self.stack_name, self.params)
+
+
+class CloudFormationDeleteStackOperator(BaseCloudFormationOperator):
+"""
+An operator that deletes a CloudFormation stack.
+
+:param stack_name: stack name.
+:type params: str
+:param params: parameters to be passed to CloudFormation. For possible
arguments see:
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.delete_stack
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r369971557
##
File path: airflow/providers/amazon/aws/operators/cloud_formation.py
##
@@ -0,0 +1,125 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains CloudFormation create/delete stack operators.
+"""
+from typing import List
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.utils.decorators import apply_defaults
+
+
+class BaseCloudFormationOperator(BaseOperator):
+"""
+Base operator for CloudFormation operations.
+
+:param params: parameters to be passed to CloudFormation.
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#1d472b'
+ui_fgcolor = '#FFF'
+
+@apply_defaults
+def __init__(
+self,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(*args, **kwargs)
+self.params = params
+self.aws_conn_id = aws_conn_id
+
+def execute(self, context):
+self.log.info('Parameters: %s', self.params)
+
+
self.cloudformation_op(AWSCloudFormationHook(aws_conn_id=self.aws_conn_id))
+
+def cloudformation_op(self, cloudformation):
+"""
+This is the main method to run CloudFormation operation.
+"""
+raise NotImplementedError()
+
+
+class CloudFormationCreateStackOperator(BaseCloudFormationOperator):
+"""
+An operator that creates a CloudFormation stack.
+
+:param stack_name: stack name.
+:type params: str
+:param params: parameters to be passed to CloudFormation. For possible
arguments see:
+
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.create_stack
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers URL: https://github.com/apache/airflow/pull/6824#discussion_r363695698 ## File path: airflow/providers/amazon/aws/sensors/cloud_formation.py ## @@ -0,0 +1,162 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +This module contains sensors for AWS CloudFormation. +""" +from botocore.exceptions import ClientError + +from airflow.providers.amazon.aws.hooks.cloud_formation import AWSCloudFormationHook +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + + +class BaseCloudFormationSensor(BaseSensorOperator): +""" +Waits for a stack operation to complete on AWS CloudFormation. + +:param stack_name: The name of the stack to wait for (templated) +:type stack_name: str +:param aws_conn_id: ID of the Airflow connection where credentials and extra configuration are +stored +:type aws_conn_id: str +:param poke_interval: Time in seconds that the job should wait between each try +:type poke_interval: int +""" + +@apply_defaults +def __init__(self, + stack_name, + complete_status, + in_progress_status, + aws_conn_id='aws_default', + poke_interval=60 * 3, + *args, + **kwargs): +super().__init__(poke_interval=poke_interval, *args, **kwargs) +self.aws_conn_id = aws_conn_id +self.stack_name = stack_name +self.complete_status = complete_status +self.in_progress_status = in_progress_status +self.hook = None + +def poke(self, context): +""" +Checks for existence of the stack in AWS CloudFormation. +""" +cloudformation = self.get_hook().get_conn() Review comment: This will create a new hook every time poke is called, is that something that is desired? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers URL: https://github.com/apache/airflow/pull/6824#discussion_r363695698 ## File path: airflow/providers/amazon/aws/sensors/cloud_formation.py ## @@ -0,0 +1,162 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +This module contains sensors for AWS CloudFormation. +""" +from botocore.exceptions import ClientError + +from airflow.providers.amazon.aws.hooks.cloud_formation import AWSCloudFormationHook +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + + +class BaseCloudFormationSensor(BaseSensorOperator): +""" +Waits for a stack operation to complete on AWS CloudFormation. + +:param stack_name: The name of the stack to wait for (templated) +:type stack_name: str +:param aws_conn_id: ID of the Airflow connection where credentials and extra configuration are +stored +:type aws_conn_id: str +:param poke_interval: Time in seconds that the job should wait between each try +:type poke_interval: int +""" + +@apply_defaults +def __init__(self, + stack_name, + complete_status, + in_progress_status, + aws_conn_id='aws_default', + poke_interval=60 * 3, + *args, + **kwargs): +super().__init__(poke_interval=poke_interval, *args, **kwargs) +self.aws_conn_id = aws_conn_id +self.stack_name = stack_name +self.complete_status = complete_status +self.in_progress_status = in_progress_status +self.hook = None + +def poke(self, context): +""" +Checks for existence of the stack in AWS CloudFormation. +""" +cloudformation = self.get_hook().get_conn() Review comment: This will create a new hook and conn every time poke is called, is that something that is desired? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r363645820
##
File path: airflow/providers/amazon/aws/operators/cloud_formation.py
##
@@ -0,0 +1,115 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains CloudFormation create/delete stack operators.
+"""
+from typing import List
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.utils.decorators import apply_defaults
+
+
+class BaseCloudFormationOperator(BaseOperator):
+"""
+Base operator for CloudFormation operations.
+
+:param params: parameters to be passed to CloudFormation.
+:type params: dict
+:param aws_conn_id: aws connection to uses
+:type aws_conn_id: str
+"""
+template_fields: List[str] = []
+template_ext = ()
+ui_color = '#1d472b'
+ui_fgcolor = '#FFF'
+
+@apply_defaults
+def __init__(
+self,
+params,
+aws_conn_id='aws_default',
+*args, **kwargs):
+super().__init__(*args, **kwargs)
+self.params = params
+self.aws_conn_id = aws_conn_id
+
+def execute(self, context):
+self.log.info('Parameters: %s', self.params)
+
+
self.cloudformation_op(AWSCloudFormationHook(aws_conn_id=self.aws_conn_id).get_conn())
+
+def cloudformation_op(self, cloudformation):
+"""
+This is the main method to run CloudFormation operation.
+"""
+raise NotImplementedError()
Review comment:
Sure, will remove the superclass.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers URL: https://github.com/apache/airflow/pull/6824#discussion_r363645551 ## File path: airflow/providers/amazon/aws/sensors/cloud_formation.py ## @@ -0,0 +1,162 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +This module contains sensors for AWS CloudFormation. +""" +from botocore.exceptions import ClientError + +from airflow.providers.amazon.aws.hooks.cloud_formation import AWSCloudFormationHook +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + + +class BaseCloudFormationSensor(BaseSensorOperator): +""" +Waits for a stack operation to complete on AWS CloudFormation. + +:param stack_name: The name of the stack to wait for (templated) +:type stack_name: str +:param aws_conn_id: ID of the Airflow connection where credentials and extra configuration are +stored +:type aws_conn_id: str +:param poke_interval: Time in seconds that the job should wait between each try +:type poke_interval: int +""" + +@apply_defaults +def __init__(self, + stack_name, + complete_status, + in_progress_status, Review comment: No, will be removed when getting rid of this superclass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r363645243
##
File path: airflow/providers/amazon/aws/sensors/cloud_formation.py
##
@@ -0,0 +1,162 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains sensors for AWS CloudFormation.
+"""
+from botocore.exceptions import ClientError
+
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class BaseCloudFormationSensor(BaseSensorOperator):
+"""
+Waits for a stack operation to complete on AWS CloudFormation.
+
+:param stack_name: The name of the stack to wait for (templated)
+:type stack_name: str
+:param aws_conn_id: ID of the Airflow connection where credentials and
extra configuration are
+stored
+:type aws_conn_id: str
+:param poke_interval: Time in seconds that the job should wait between
each try
+:type poke_interval: int
+"""
+
+@apply_defaults
+def __init__(self,
+ stack_name,
+ complete_status,
+ in_progress_status,
+ aws_conn_id='aws_default',
+ poke_interval=60 * 3,
+ *args,
+ **kwargs):
+super().__init__(poke_interval=poke_interval, *args, **kwargs)
+self.aws_conn_id = aws_conn_id
+self.stack_name = stack_name
+self.complete_status = complete_status
+self.in_progress_status = in_progress_status
+self.hook = None
+
+def poke(self, context):
+"""
+Checks for existence of the stack in AWS CloudFormation.
+"""
+cloudformation = self.get_hook().get_conn()
+
+self.log.info('Poking for stack %s', self.stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=self.stack_name)['Stacks']
+stack_status = stacks[0]['StackStatus']
+if stack_status == self.complete_status:
+return True
+elif stack_status == self.in_progress_status:
+return False
+else:
+raise ValueError(f'Stack {self.stack_name} in bad state:
{stack_status}')
+except ClientError as e:
+if 'does not exist' in str(e):
+if not self.allow_non_existing_stack_status():
+raise ValueError(f'Stack {self.stack_name} does not exist')
+else:
+return True
+else:
+raise e
+
+def get_hook(self):
+"""
+Gets the AwsGlueCatalogHook
+"""
+if not self.hook:
+self.hook = AWSCloudFormationHook(aws_conn_id=self.aws_conn_id)
+
+return self.hook
+
+def allow_non_existing_stack_status(self):
+"""
+Boolean value whether or not sensor should allow non existing stack
responses.
+"""
+return False
+
+
+class CloudFormationCreateStackSensor(BaseCloudFormationSensor):
+"""
+Waits for a stack to be created successfully on AWS CloudFormation.
+
+:param stack_name: The name of the stack to wait for (templated)
+:type stack_name: str
+:param aws_conn_id: ID of the Airflow connection where credentials and
extra configuration are
+stored
+:type aws_conn_id: str
+:param poke_interval: Time in seconds that the job should wait between
each try
+:type poke_interval: int
+"""
+
+template_fields = ['stack_name']
+ui_color = '#C5CAE9'
+
+@apply_defaults
+def __init__(self,
+ stack_name,
+ aws_conn_id='aws_default',
+ poke_interval=60 * 3,
+ *args,
+ **kwargs):
+super().__init__(stack_name=stack_name,
+ complete_status='CREATE_COMPLETE',
+ in_progress_status='CREATE_IN_PROGRESS',
+ aws_conn_id=aws_conn_id,
+
[GitHub] [airflow] aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add CloudFormation operators to AWS providers
aviemzur commented on a change in pull request #6824: [AIRFLOW-6258] add
CloudFormation operators to AWS providers
URL: https://github.com/apache/airflow/pull/6824#discussion_r363644240
##
File path: airflow/providers/amazon/aws/sensors/cloud_formation.py
##
@@ -0,0 +1,162 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains sensors for AWS CloudFormation.
+"""
+from botocore.exceptions import ClientError
+
+from airflow.providers.amazon.aws.hooks.cloud_formation import
AWSCloudFormationHook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class BaseCloudFormationSensor(BaseSensorOperator):
+"""
+Waits for a stack operation to complete on AWS CloudFormation.
+
+:param stack_name: The name of the stack to wait for (templated)
+:type stack_name: str
+:param aws_conn_id: ID of the Airflow connection where credentials and
extra configuration are
+stored
+:type aws_conn_id: str
+:param poke_interval: Time in seconds that the job should wait between
each try
+:type poke_interval: int
+"""
+
+@apply_defaults
+def __init__(self,
+ stack_name,
+ complete_status,
+ in_progress_status,
+ aws_conn_id='aws_default',
+ poke_interval=60 * 3,
+ *args,
+ **kwargs):
+super().__init__(poke_interval=poke_interval, *args, **kwargs)
+self.aws_conn_id = aws_conn_id
+self.stack_name = stack_name
+self.complete_status = complete_status
+self.in_progress_status = in_progress_status
+self.hook = None
+
+def poke(self, context):
+"""
+Checks for existence of the stack in AWS CloudFormation.
+"""
+cloudformation = self.get_hook().get_conn()
+
+self.log.info('Poking for stack %s', self.stack_name)
+
+try:
+stacks =
cloudformation.describe_stacks(StackName=self.stack_name)['Stacks']
+stack_status = stacks[0]['StackStatus']
+if stack_status == self.complete_status:
+return True
+elif stack_status == self.in_progress_status:
+return False
+else:
+raise ValueError(f'Stack {self.stack_name} in bad state:
{stack_status}')
+except ClientError as e:
+if 'does not exist' in str(e):
+if not self.allow_non_existing_stack_status():
+raise ValueError(f'Stack {self.stack_name} does not exist')
+else:
+return True
+else:
+raise e
+
+def get_hook(self):
+"""
+Gets the AwsGlueCatalogHook
+"""
+if not self.hook:
+self.hook = AWSCloudFormationHook(aws_conn_id=self.aws_conn_id)
+
+return self.hook
+
+def allow_non_existing_stack_status(self):
+"""
+Boolean value whether or not sensor should allow non existing stack
responses.
+"""
+return False
Review comment:
When you query CloudFormation about a stack and it doesn't exist it throws
an exception.
For checking if a cluster was created and you get this exception, that means
something went wrong.
For checking if a cluster was deleted, this sadly could be an appropriate
response from CloudFormation API, the cluster has indeed been deleted.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
