[jira] [Commented] (AIRFLOW-2826) Add hook for Google Cloud KMS

2018-08-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572847#comment-16572847
 ] 

ASF GitHub Bot commented on AIRFLOW-2826:
-

Fokko closed pull request #3677: [AIRFLOW-2826] Add GoogleCloudKMSHook
URL: https://github.com/apache/incubator-airflow/pull/3677
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/contrib/hooks/gcp_kms_hook.py 
b/airflow/contrib/hooks/gcp_kms_hook.py
new file mode 100644
index 00..6f2b3aedff
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_kms_hook.py
@@ -0,0 +1,108 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import base64
+
+from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+
+from apiclient.discovery import build
+
+
+def _b64encode(s):
+""" Base 64 encodes a bytes object to a string """
+return base64.b64encode(s).decode('ascii')
+
+
+def _b64decode(s):
+""" Base 64 decodes a string to bytes. """
+return base64.b64decode(s.encode('utf-8'))
+
+
+class GoogleCloudKMSHook(GoogleCloudBaseHook):
+"""
+Interact with Google Cloud KMS. This hook uses the Google Cloud Platform
+connection.
+"""
+
+def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+super(GoogleCloudKMSHook, self).__init__(gcp_conn_id, 
delegate_to=delegate_to)
+
+def get_conn(self):
+"""
+Returns a KMS service object.
+
+:rtype: apiclient.discovery.Resource
+"""
+http_authorized = self._authorize()
+return build(
+'cloudkms', 'v1', http=http_authorized, cache_discovery=False)
+
+def encrypt(self, key_name, plaintext, authenticated_data=None):
+"""
+Encrypts a plaintext message using Google Cloud KMS.
+
+:param key_name: The Resource Name for the key (or key version)
+ to be used for encyption. Of the form
+ ``projects/*/locations/*/keyRings/*/cryptoKeys/**``
+:type key_name: str
+:param plaintext: The message to be encrypted.
+:type plaintext: bytes
+:param authenticated_data: Optional additional authenticated data that
+   must also be provided to decrypt the 
message.
+:type authenticated_data: bytes
+:return: The base 64 encoded ciphertext of the original message.
+:rtype: str
+"""
+keys = self.get_conn().projects().locations().keyRings().cryptoKeys()
+body = {'plaintext': _b64encode(plaintext)}
+if authenticated_data:
+body['additionalAuthenticatedData'] = 
_b64encode(authenticated_data)
+
+request = keys.encrypt(name=key_name, body=body)
+response = request.execute()
+
+ciphertext = response['ciphertext']
+return ciphertext
+
+def decrypt(self, key_name, ciphertext, authenticated_data=None):
+"""
+Decrypts a ciphertext message using Google Cloud KMS.
+
+:param key_name: The Resource Name for the key to be used for 
decyption.
+ Of the form 
``projects/*/locations/*/keyRings/*/cryptoKeys/**``
+:type key_name: str
+:param ciphertext: The message to be decrypted.
+:type ciphertext: str
+:param authenticated_data: Any additional authenticated data that was
+   provided when encrypting the message.
+:type authenticated_data: bytes
+:return: The original message.
+:rtype: bytes
+"""
+keys = self.get_conn().projects().locations().keyRings().cryptoKeys()
+body = {'ciphertext': ciphertext}
+if authenticated_data:
+body['additionalAuthenticatedData'] = 
_b64encode(authenticated_data)
+
+request = keys.decrypt(name=key_name, body=body)
+response = request.execute()
+
+plaintext = 

[jira] [Commented] (AIRFLOW-2826) Add hook for Google Cloud KMS

2018-08-08 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572848#comment-16572848
 ] 

ASF subversion and git services commented on AIRFLOW-2826:
--

Commit acca61c602e341da06ebee2eca3a26f4e7400238 in incubator-airflow's branch 
refs/heads/master from [~jakahn]
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=acca61c ]

[AIRFLOW-2826] Add GoogleCloudKMSHook (#3677)

Adds a hook enabling encryption and decryption through Google Cloud KMS.
This should also contribute to AIRFLOW-2062.

> Add hook for Google Cloud KMS
> -
>
> Key: AIRFLOW-2826
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2826
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Reporter: Jasper Kahn
>Assignee: Jasper Kahn
>Priority: Minor
>  Labels: features
>
> Add a hook to support interacting with Google Cloud KMS. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (AIRFLOW-2826) Add hook for Google Cloud KMS

2018-08-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566064#comment-16566064
 ] 

ASF GitHub Bot commented on AIRFLOW-2826:
-

jakahn opened a new pull request #3677: [AIRFLOW-2826] Add GoogleCloudKMSHook
URL: https://github.com/apache/incubator-airflow/pull/3677
 
 
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title
 - https://issues.apache.org/jira/browse/AIRFLOW-2826
   
   ### Description
   
   - [X] Here are some details about my PR (no UI changes):
  - Adds a hook enabling encryption and decryption through Google Cloud 
KMS. Both operations also support the use of the "additionalAuthenticatedData" 
field for the requests.
  - This hook is also added in anticipation of integration with future work 
on [AIRFLOW-2062](https://issues.apache.org/jira/browse/AIRFLOW-2062).
   
   ### Tests
   
   - [X] My PR adds the following unit tests:
  - `tests/contrib/hooks/test_gcp_kms_hook.py`
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [X] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add hook for Google Cloud KMS
> -
>
> Key: AIRFLOW-2826
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2826
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Reporter: Jasper Kahn
>Assignee: Jasper Kahn
>Priority: Minor
>  Labels: features
>
> Add a hook to support interacting with Google Cloud KMS. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)