[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151856#comment-17151856 ] ASF GitHub Bot commented on PARQUET-1373: - gszadovszky merged pull request #615: URL: https://github.com/apache/parquet-mr/pull/615 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149592#comment-17149592 ] Jason Brugger commented on PARQUET-1373: unsubscribe On Wed, Jul 1, 2020 at 9:31 AM ASF GitHub Bot (Jira) > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149477#comment-17149477 ] ASF GitHub Bot commented on PARQUET-1373: - gszadovszky closed pull request #615: URL: https://github.com/apache/parquet-mr/pull/615 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149476#comment-17149476 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky opened a new pull request #615: URL: https://github.com/apache/parquet-mr/pull/615 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149252#comment-17149252 ] ASF GitHub Bot commented on PARQUET-1373: - gszadovszky commented on a change in pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#discussion_r448244268 ## File path: parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/KeyMaterial.java ## @@ -28,14 +28,35 @@ import org.codehaus.jackson.map.ObjectMapper; import org.codehaus.jackson.type.TypeReference; +/** + * KeyMaterial class represents the "key material", keeping the information that allows readers to recover an encryption key (see + * description of the KeyMetadata class). The keytools package (PARQUET-1373) implements the "envelope encryption" pattern, in a + * "single wrapping" or "double wrapping" mode. In the single wrapping mode, the key material is generated by encrypting the + * "data encryption key" (DEK) by a "master key". In the double wrapping mode, the key material is generated by encrypting the DEK + * by a "key encryption key" (KEK), that in turn is encrypted by a "master key". + * + * Key material is kept in a flat json object, with the following fields: + * 1. "keyMaterialType" - a String, with the type of key material. In the current version, only one value is allowed - "PKMT1" (stands + * for "parquet key management tools, version 1"). For external key material storage, this field is written in both "key metadata" and + * "key material" jsons. For internal key material storage, this field is written only once in the common json. + * 2. "isFooterKey" - a boolean. If true, means that the material belongs to a file footer key, and keeps additional information (such as Review comment: This (and the other _"boolean"_ members) are not `boolean` values in the as per the json spec. (You are writing `"true"` and `"false"` instead of `true` and `false`.) You may either mention it in the description or try to use a `Map` and put `String` or `Boolean` values. I think, `jackson` will write proper `boolean` values in this case. I would vote on the latter if it works. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149234#comment-17149234 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky commented on pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#issuecomment-652301855 @gszadovszky the latest review round is addressed by 0355f65 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148485#comment-17148485
]
ASF GitHub Bot commented on PARQUET-1373:
-
gszadovszky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r447561609
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/KeyMaterial.java
##
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.crypto.keytools;
+
+import java.io.IOException;
+import java.io.StringReader;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.codehaus.jackson.type.TypeReference;
+
+public class KeyMaterial {
+ static final String KEY_MATERIAL_TYPE_FIELD = "keyMaterialType";
+ static final String KEY_MATERIAL_TYPE = "PKMT1";
+ static final String KEY_MATERIAL_INTERNAL_STORAGE_FIELD = "internalStorage";
+
+ static final String FOOTER_KEY_ID_IN_FILE = "footerKey";
+ static final String COLUMN_KEY_ID_IN_FILE_PREFIX = "columnKey";
+
+ private static final String IS_FOOTER_KEY_FIELD = "isFooterKey";
+ private static final String DOUBLE_WRAPPING_FIELD = "doubleWrapping";
+ private static final String KMS_INSTANCE_ID_FIELD = "kmsInstanceID";
+ private static final String KMS_INSTANCE_URL_FIELD = "kmsInstanceURL";
+ private static final String MASTER_KEY_ID_FIELD = "masterKeyID";
+ private static final String WRAPPED_DEK_FIELD = "wrappedDEK";
+ private static final String KEK_ID_FIELD = "keyEncryptionKeyID";
+ private static final String WRAPPED_KEK_FIELD = "wrappedKEK";
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ private final boolean isFooterKey;
+ private final String kmsInstanceID;
+ private final String kmsInstanceURL;
+ private final String masterKeyID;
+ private final boolean isDoubleWrapped;
+ private final String kekID;
+ private final String encodedWrappedKEK;
+ private final String encodedWrappedDEK;
+
+ private KeyMaterial(boolean isFooterKey, String kmsInstanceID, String
kmsInstanceURL, String masterKeyID,
+ boolean isDoubleWrapped, String kekID, String encodedWrappedKEK, String
encodedWrappedDEK) {
+this.isFooterKey = isFooterKey;
+this.kmsInstanceID = kmsInstanceID;
+this.kmsInstanceURL = kmsInstanceURL;
+this.masterKeyID = masterKeyID;
+this.isDoubleWrapped = isDoubleWrapped;
+this.kekID = kekID;
+this.encodedWrappedKEK = encodedWrappedKEK;
+this.encodedWrappedDEK = encodedWrappedDEK;
+ }
+
+ static KeyMaterial parse(Map keyMaterialJson) {
+boolean isFooterKey =
Boolean.valueOf(keyMaterialJson.get(IS_FOOTER_KEY_FIELD));
+String kmsInstanceID = null;
+String kmsInstanceURL = null;
+if (isFooterKey) {
+ kmsInstanceID = keyMaterialJson.get(KMS_INSTANCE_ID_FIELD);
+ kmsInstanceURL = keyMaterialJson.get(KMS_INSTANCE_URL_FIELD);
+}
+boolean isDoubleWrapped =
Boolean.valueOf(keyMaterialJson.get(DOUBLE_WRAPPING_FIELD));
+String masterKeyID = keyMaterialJson.get(MASTER_KEY_ID_FIELD);
+String encodedWrappedDEK = keyMaterialJson.get(WRAPPED_DEK_FIELD);
+String kekID = null;
+String encodedWrappedKEK = null;
+if (isDoubleWrapped) {
+ kekID = keyMaterialJson.get(KEK_ID_FIELD);
+ encodedWrappedKEK = keyMaterialJson.get(WRAPPED_KEK_FIELD);
+}
+
+return new KeyMaterial(isFooterKey, kmsInstanceID, kmsInstanceURL,
masterKeyID, isDoubleWrapped, kekID, encodedWrappedKEK, encodedWrappedDEK);
+ }
+
+ static KeyMaterial parse(String keyMaterialString) {
+Map keyMaterialJson = null;
+try {
+ keyMaterialJson = OBJECT_MAPPER.readValue(new
StringReader(keyMaterialString),
+ new TypeReference>() {});
+} catch (IOException e) {
+ throw new ParquetCryptoRuntimeException("Failed to parse key metadata "
+ keyMaterialString, e);
+}
+String keyMaterialType = keyMaterialJson.get(KEY_MATERIAL_TYPE_FIELD);
+if (!KEY_MATERIAL_TYPE.equals(keyMaterialType)) {
+
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148442#comment-17148442 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky commented on a change in pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#discussion_r447508802 ## File path: parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/KeyMaterial.java ## @@ -0,0 +1,166 @@ +/* Review comment: > .. With the correct annotations it can map a java object automatically. This approach seems to be optimal for objects with a fixed structure / fields - because it searches for all object fields in the json file. In our case, many fields are not always written (eg kms instance or url, if the key is for a column, and not for the footer; and other examples). Searching for them always is an overhead. Moreover, in case of internal storage, we don't need to parse two objects - key metadata and key material, because they are the same, so parsing one object is sufficient. The code we have today, performs only the search/parse of the relevant objects/fields, so it is optimal in that sense - and is also well-defined in one place. We can add more comments to the code to make the field parsing logic crystal clear. What do you think? > > the format of these json objects is important for compatibility. We shall specify them or at least give an example in the comments. Sounds good. In addition to the comments mentioned above (that will be added to the relevant code lines), we will add a class comment to each relevant class, that documents the structure of the corresponding json. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148437#comment-17148437
]
ASF GitHub Bot commented on PARQUET-1373:
-
ggershinsky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r447509073
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/PropertiesDrivenCryptoFactory.java
##
@@ -36,38 +39,62 @@
import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
import org.apache.parquet.hadoop.api.WriteSupport.WriteContext;
import org.apache.parquet.hadoop.metadata.ColumnPath;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import static org.apache.parquet.crypto.keytools.KeyToolkit.stringIsEmpty;
public class PropertiesDrivenCryptoFactory implements
EncryptionPropertiesFactory, DecryptionPropertiesFactory {
-
- public static final String COLUMN_KEYS_PROPERTY_NAME =
"encryption.column.keys";
- public static final String FOOTER_KEY_PROPERTY_NAME =
"encryption.footer.key";
- public static final String ENCRYPTION_ALGORITHM_PROPERTY_NAME =
"encryption.algorithm";
- public static final String PLAINTEXT_FOOTER_PROPERTY_NAME =
"encryption.plaintext.footer";
+ private static final Logger LOG =
LoggerFactory.getLogger(PropertiesDrivenCryptoFactory.class);
- public static final int DEK_LENGTH = 16;
-
- private static final SecureRandom random = new SecureRandom();
+ private static final Integer[] ACCEPTABLE_DATA_KEY_LENGTHS = {128, 192, 256};
+ private static final Set ACCEPTABLE_DATA_KEY_LENGTHS_SET =
+new HashSet<>(Arrays.asList(ACCEPTABLE_DATA_KEY_LENGTHS));
Review comment:
Sure, we'll change this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Encryption key management tools
>
>
> Key: PARQUET-1373
> URL: https://issues.apache.org/jira/browse/PARQUET-1373
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-mr
>Reporter: Gidon Gershinsky
>Assignee: Gidon Gershinsky
>Priority: Major
>
> Parquet Modular Encryption
> ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides
> an API that accepts keys, arbitrary key metadata and key retrieval callbacks
> - which allows to implement basically any key management policy on top of it.
> This Jira will add tools that implement a set of best practice elements for
> key management. This is not an end-to-end key management, but rather a set of
> components that might simplify design and development of an end-to-end
> solution.
> This tool set is one of many possible. There is no goal to create a single or
> “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory
> interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817])
> that allows to plug in different implementations of encryption key management.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148436#comment-17148436 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky commented on a change in pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#discussion_r447508802 ## File path: parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/KeyMaterial.java ## @@ -0,0 +1,166 @@ +/* Review comment: > .. With the correct annotations it can map a java object automatically. This approach seems to be optimal for objects with a fixed structure / fields - because it searches for all object fields in the json file. In our case, many fields are not always written (eg kms instance or url, if the key is for a column, and not for the footer; and other examples). Searching for them always is an overhead. Moreover, in case of internal storage, we don't need to parse two objects - key metadata and key material, because they are the same, so parsing one object is sufficient. The code we have today, performs only the search/parse of the relevant objects/fields, so it is optimal in that sense - and also well-defined in one place. We can add more comments to the code to make the field parsing logic crystal clear. What do you think? > > I think, the format of these json objects is important for compatibility. We shall specify them or at least give an example in the comments. Sounds good. In addition to the comments mentioned above (that will be added to the relevant code lines), we will add a class comment to each relevant class, that documents the structure of the corresponding json. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148434#comment-17148434
]
ASF GitHub Bot commented on PARQUET-1373:
-
ggershinsky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r447508660
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/KeyMaterial.java
##
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.crypto.keytools;
+
+import java.io.IOException;
+import java.io.StringReader;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.codehaus.jackson.type.TypeReference;
+
+public class KeyMaterial {
+ static final String KEY_MATERIAL_TYPE_FIELD = "keyMaterialType";
+ static final String KEY_MATERIAL_TYPE = "PKMT1";
+ static final String KEY_MATERIAL_INTERNAL_STORAGE_FIELD = "internalStorage";
+
+ static final String FOOTER_KEY_ID_IN_FILE = "footerKey";
+ static final String COLUMN_KEY_ID_IN_FILE_PREFIX = "columnKey";
+
+ private static final String IS_FOOTER_KEY_FIELD = "isFooterKey";
+ private static final String DOUBLE_WRAPPING_FIELD = "doubleWrapping";
+ private static final String KMS_INSTANCE_ID_FIELD = "kmsInstanceID";
+ private static final String KMS_INSTANCE_URL_FIELD = "kmsInstanceURL";
+ private static final String MASTER_KEY_ID_FIELD = "masterKeyID";
+ private static final String WRAPPED_DEK_FIELD = "wrappedDEK";
+ private static final String KEK_ID_FIELD = "keyEncryptionKeyID";
+ private static final String WRAPPED_KEK_FIELD = "wrappedKEK";
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ private final boolean isFooterKey;
+ private final String kmsInstanceID;
+ private final String kmsInstanceURL;
+ private final String masterKeyID;
+ private final boolean isDoubleWrapped;
+ private final String kekID;
+ private final String encodedWrappedKEK;
+ private final String encodedWrappedDEK;
+
+ private KeyMaterial(boolean isFooterKey, String kmsInstanceID, String
kmsInstanceURL, String masterKeyID,
+ boolean isDoubleWrapped, String kekID, String encodedWrappedKEK, String
encodedWrappedDEK) {
+this.isFooterKey = isFooterKey;
+this.kmsInstanceID = kmsInstanceID;
+this.kmsInstanceURL = kmsInstanceURL;
+this.masterKeyID = masterKeyID;
+this.isDoubleWrapped = isDoubleWrapped;
+this.kekID = kekID;
+this.encodedWrappedKEK = encodedWrappedKEK;
+this.encodedWrappedDEK = encodedWrappedDEK;
+ }
+
+ static KeyMaterial parse(Map keyMaterialJson) {
+boolean isFooterKey =
Boolean.valueOf(keyMaterialJson.get(IS_FOOTER_KEY_FIELD));
+String kmsInstanceID = null;
+String kmsInstanceURL = null;
+if (isFooterKey) {
+ kmsInstanceID = keyMaterialJson.get(KMS_INSTANCE_ID_FIELD);
+ kmsInstanceURL = keyMaterialJson.get(KMS_INSTANCE_URL_FIELD);
+}
+boolean isDoubleWrapped =
Boolean.valueOf(keyMaterialJson.get(DOUBLE_WRAPPING_FIELD));
+String masterKeyID = keyMaterialJson.get(MASTER_KEY_ID_FIELD);
+String encodedWrappedDEK = keyMaterialJson.get(WRAPPED_DEK_FIELD);
+String kekID = null;
+String encodedWrappedKEK = null;
+if (isDoubleWrapped) {
+ kekID = keyMaterialJson.get(KEK_ID_FIELD);
+ encodedWrappedKEK = keyMaterialJson.get(WRAPPED_KEK_FIELD);
+}
+
+return new KeyMaterial(isFooterKey, kmsInstanceID, kmsInstanceURL,
masterKeyID, isDoubleWrapped, kekID, encodedWrappedKEK, encodedWrappedDEK);
+ }
+
+ static KeyMaterial parse(String keyMaterialString) {
+Map keyMaterialJson = null;
+try {
+ keyMaterialJson = OBJECT_MAPPER.readValue(new
StringReader(keyMaterialString),
+ new TypeReference>() {});
+} catch (IOException e) {
+ throw new ParquetCryptoRuntimeException("Failed to parse key metadata "
+ keyMaterialString, e);
+}
+String keyMaterialType = keyMaterialJson.get(KEY_MATERIAL_TYPE_FIELD);
+if (!KEY_MATERIAL_TYPE.equals(keyMaterialType)) {
+
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147824#comment-17147824
]
ASF GitHub Bot commented on PARQUET-1373:
-
gszadovszky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r446146320
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/KeyMaterial.java
##
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.crypto.keytools;
+
+import java.io.IOException;
+import java.io.StringReader;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.codehaus.jackson.type.TypeReference;
+
+public class KeyMaterial {
+ static final String KEY_MATERIAL_TYPE_FIELD = "keyMaterialType";
+ static final String KEY_MATERIAL_TYPE = "PKMT1";
+ static final String KEY_MATERIAL_INTERNAL_STORAGE_FIELD = "internalStorage";
+
+ static final String FOOTER_KEY_ID_IN_FILE = "footerKey";
+ static final String COLUMN_KEY_ID_IN_FILE_PREFIX = "columnKey";
+
+ private static final String IS_FOOTER_KEY_FIELD = "isFooterKey";
+ private static final String DOUBLE_WRAPPING_FIELD = "doubleWrapping";
+ private static final String KMS_INSTANCE_ID_FIELD = "kmsInstanceID";
+ private static final String KMS_INSTANCE_URL_FIELD = "kmsInstanceURL";
+ private static final String MASTER_KEY_ID_FIELD = "masterKeyID";
+ private static final String WRAPPED_DEK_FIELD = "wrappedDEK";
+ private static final String KEK_ID_FIELD = "keyEncryptionKeyID";
+ private static final String WRAPPED_KEK_FIELD = "wrappedKEK";
+
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ private final boolean isFooterKey;
+ private final String kmsInstanceID;
+ private final String kmsInstanceURL;
+ private final String masterKeyID;
+ private final boolean isDoubleWrapped;
+ private final String kekID;
+ private final String encodedWrappedKEK;
+ private final String encodedWrappedDEK;
+
+ private KeyMaterial(boolean isFooterKey, String kmsInstanceID, String
kmsInstanceURL, String masterKeyID,
+ boolean isDoubleWrapped, String kekID, String encodedWrappedKEK, String
encodedWrappedDEK) {
+this.isFooterKey = isFooterKey;
+this.kmsInstanceID = kmsInstanceID;
+this.kmsInstanceURL = kmsInstanceURL;
+this.masterKeyID = masterKeyID;
+this.isDoubleWrapped = isDoubleWrapped;
+this.kekID = kekID;
+this.encodedWrappedKEK = encodedWrappedKEK;
+this.encodedWrappedDEK = encodedWrappedDEK;
+ }
+
+ static KeyMaterial parse(Map keyMaterialJson) {
+boolean isFooterKey =
Boolean.valueOf(keyMaterialJson.get(IS_FOOTER_KEY_FIELD));
+String kmsInstanceID = null;
+String kmsInstanceURL = null;
+if (isFooterKey) {
+ kmsInstanceID = keyMaterialJson.get(KMS_INSTANCE_ID_FIELD);
+ kmsInstanceURL = keyMaterialJson.get(KMS_INSTANCE_URL_FIELD);
+}
+boolean isDoubleWrapped =
Boolean.valueOf(keyMaterialJson.get(DOUBLE_WRAPPING_FIELD));
+String masterKeyID = keyMaterialJson.get(MASTER_KEY_ID_FIELD);
+String encodedWrappedDEK = keyMaterialJson.get(WRAPPED_DEK_FIELD);
+String kekID = null;
+String encodedWrappedKEK = null;
+if (isDoubleWrapped) {
+ kekID = keyMaterialJson.get(KEK_ID_FIELD);
+ encodedWrappedKEK = keyMaterialJson.get(WRAPPED_KEK_FIELD);
+}
+
+return new KeyMaterial(isFooterKey, kmsInstanceID, kmsInstanceURL,
masterKeyID, isDoubleWrapped, kekID, encodedWrappedKEK, encodedWrappedDEK);
+ }
+
+ static KeyMaterial parse(String keyMaterialString) {
+Map keyMaterialJson = null;
+try {
+ keyMaterialJson = OBJECT_MAPPER.readValue(new
StringReader(keyMaterialString),
+ new TypeReference>() {});
+} catch (IOException e) {
+ throw new ParquetCryptoRuntimeException("Failed to parse key metadata "
+ keyMaterialString, e);
+}
+String keyMaterialType = keyMaterialJson.get(KEY_MATERIAL_TYPE_FIELD);
+if (!KEY_MATERIAL_TYPE.equals(keyMaterialType)) {
+
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144968#comment-17144968 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky commented on pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#issuecomment-649588181 @gszadovszky the 055bb39 is the squash of all commits that address the latest review round This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136555#comment-17136555 ] ASF GitHub Bot commented on PARQUET-1373: - gszadovszky commented on pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#issuecomment-644698810 Thanks a lot for explanation. Agreed on protecting sensitive data in memory cannot be handled by Parquet. I'll wait for your comment for the next review round. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135906#comment-17135906 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky edited a comment on pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#issuecomment-644145175 > > ConcurrentMap has a segment synchronization for write operations, and allows for synchronization-free read operations; this makes it faster than HasMap with synchronized methods. > > Yes, I know how `ConcurrentHashMap` works. What I wanted to say that you are using synchronization as well. As you already use a `ConcurrentMap` you might implement these synchronized code parts by using the methods of `ConcurrentMap`. I've put some examples that might work. Sounds good, and thank you for the examples! We've already applied your code (not pushed yet), it indeed allowed to remove the explicit synchronization, making the cache implementation cleaner and faster. > > Please, check why Travis fails. > Sorry, should have mentioned that it will take a few more commits to fully address this round of the comments (and fix the unitests). Once all commits are in, I will squash them to simplify the review, and will post a comment here. > Another point of view came up about handling sensitive data in memory. Java does not clean memory after garbage collecting objects. It means that sensitive data must be manually cleaned after used otherwise it might get compromised by another java application in the same jvm or even by another process after the jvm exists. Because of the same reason `String` objects shall never contain sensitive information as the `char[]` behind the object might not get garbage collected after the `String` object itself gets dropped. > I did not find any particular bad practice in the code or any examples of the listed situations just wanted to highlight that we shall think about this as well. Yep, keeping secret data in Java strings is a notorious problem. I think the general consensus is not to rely on gc or explicit byte wiping - but to remember that these Java processes must run in a trusted environment anyway, simply because they work with confidential information, ranging from the encryption keys to the sensitive data itself. Micro-managing the memory with confidential information is always hard, and is basically impossible with Java. It goes beyond Parquet. One example - the KMS Client implementations send secret tokens and fetch explicit encryption keys, using a custom HTTP library. There is no guarantee this library doesn't use strings (most likely, it does). Another example - the secret tokens are passed as a Hadoop property from Spark or another framework; this is likely to be implemented with strings. Moreover, the tokens are built in an access control system, then sent to a user, then sent to a Spark driver, then sent to Spark workers (or other framework components) - there is no way to control this, except to rely on HTTPS for the transport security, and on running framework drivers/workers in a trusted environment for the memory security. In other words, our threat model is simple. We don't trust the storage - encrypted Parquet files can be accessed by malicious parties, but they won't be able to read them. We do trust the framework hosts (where the JVM runs) - if these are breached, the secret data can be stolen from any part of host memory / disc pages; not just the Parquet lib memory, but framework memory, HTTP libs, etc. Memory protection is a holy grail in this field, addressed by technologies like VMs, containers, hardware enclaves, etc, etc. Parquet encryption is focused on data-in-storage protection; data-in-memory protection is covered by other technologies. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135871#comment-17135871 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky commented on pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#issuecomment-644145175 > > ConcurrentMap has a segment synchronization for write operations, and allows for synchronization-free read operations; this makes it faster than HasMap with synchronized methods. > > Yes, I know how `ConcurrentHashMap` works. What I wanted to say that you are using synchronization as well. As you already use a `ConcurrentMap` you might implement these synchronized code parts by using the methods of `ConcurrentMap`. I've put some examples that might work. Sounds good, and thank you for the examples! We've already applied your code (not pushed yet), it indeed allowed to remove the explicit synchronization, making the cache implementation cleaner and faster. > > Please, check why Travis fails. > Sorry, should have mentioned that it will take a few more commits to fully address this round of the comments (and fix the unitests). Once all commits are in, I will squash them to simplify the review, and will post a comment here. > Another point of view came up about handling sensitive data in memory. Java does not clean memory after garbage collecting objects. It means that sensitive data must be manually cleaned after used otherwise it might get compromised by another java application in the same jvm or even by another process after the jvm exists. Because of the same reason `String` objects shall never contain sensitive information as the `char[]` behind the object might not get garbage collected after the `String` object itself gets dropped. > I did not find any particular bad practice in the code or any examples of the listed situations just wanted to highlight that we shall think about this as well. Yep, keeping secret data in Java strings is a notorious problem. I think the general consensus is not to rely on gc or explicit byte wiping - but to remember that these Java processes must run in a trusted environment anyway, simply because they work with confidential information, ranging from the encryption keys to the sensitive data itself. Micro-managing the memory with confidential information is always hard, and is basically impossible with Java. It goes beyond Parquet. One example - the KMS Client implementations send secret tokens and fetch explicit encryption keys, using a custom HTTP library. There is no guarantee this library doesn't use strings (most likely, it does). Another example - the secret tokens are passed as a Hadoop property from Spark or another framework; this is likely to be implemented with strings. Moreover, the tokens are built in an access control system, then sent to a user, then sent to a Spark driver, then sent to Spark workers (or other framework components) - there is no way to control this, except to rely on HTTPS for the transport security, and on running framework drivers/workers in a trusted environment for the memory security. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135788#comment-17135788
]
ASF GitHub Bot commented on PARQUET-1373:
-
gszadovszky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r439346745
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/PropertiesDrivenCryptoFactory.java
##
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.crypto.keytools;
+
+import java.io.IOException;
+import java.security.SecureRandom;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.crypto.ColumnEncryptionProperties;
+import org.apache.parquet.crypto.DecryptionKeyRetriever;
+import org.apache.parquet.crypto.DecryptionPropertiesFactory;
+import org.apache.parquet.crypto.EncryptionPropertiesFactory;
+import org.apache.parquet.crypto.FileDecryptionProperties;
+import org.apache.parquet.crypto.FileEncryptionProperties;
+import org.apache.parquet.crypto.ParquetCipher;
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.apache.parquet.hadoop.api.WriteSupport.WriteContext;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+
+import static org.apache.parquet.crypto.keytools.KeyToolkit.stringIsEmpty;
+
+public class PropertiesDrivenCryptoFactory implements
EncryptionPropertiesFactory, DecryptionPropertiesFactory {
+
+ public static final String COLUMN_KEYS_PROPERTY_NAME =
"encryption.column.keys";
+ public static final String FOOTER_KEY_PROPERTY_NAME =
"encryption.footer.key";
+ public static final String ENCRYPTION_ALGORITHM_PROPERTY_NAME =
"encryption.algorithm";
+ public static final String PLAINTEXT_FOOTER_PROPERTY_NAME =
"encryption.plaintext.footer";
+
+ public static final int DEK_LENGTH = 16;
+
+ private static final SecureRandom random = new SecureRandom();
Review comment:
As per the java naming conventions:
```suggestion
private static final SecureRandom RANDOM = new SecureRandom();
```
##
File path:
parquet-hadoop/src/test/java/org/apache/parquet/crypto/TestPropertiesDrivenEncryption.java
##
@@ -0,0 +1,507 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.crypto;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.crypto.keytools.KeyToolkit;
+import org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory;
+import org.apache.parquet.crypto.keytools.samples.InMemoryKMS;
+import org.apache.parquet.crypto.mocks.RemoteKmsClientMock;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroupFactory;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.example.GroupReadSupport;
+import org.apache.parquet.hadoop.example.GroupWriteSupport;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Types;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ErrorCollector;
+import
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135619#comment-17135619
]
ASF GitHub Bot commented on PARQUET-1373:
-
ggershinsky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r440027472
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/HadoopFSKeyMaterialStore.java
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.crypto.keytools;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.codehaus.jackson.JsonNode;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.codehaus.jackson.type.TypeReference;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+public class HadoopFSKeyMaterialStore implements FileKeyMaterialStore {
+
+ public final static String KEY_MATERIAL_FILE_PREFIX = "_KEY_MATERIAL_FOR_";
+ public static final String TEMP_FILE_PREFIX = "_TMP";
+ public final static String KEY_MATERIAL_FILE_SUFFFIX = ".json";
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private FileSystem hadoopFileSystem;
+ private Map keyMaterialMap;
+ private Path keyMaterialFile;
+
+ HadoopFSKeyMaterialStore(FileSystem hadoopFileSystem) {
+this.hadoopFileSystem = hadoopFileSystem;
+ }
+
+ @Override
+ public void initialize(Path parquetFilePath, Configuration hadoopConfig,
boolean tempStore) {
+String fullPrefix = (tempStore? TEMP_FILE_PREFIX : "");
+fullPrefix += KEY_MATERIAL_FILE_PREFIX;
+keyMaterialFile = new Path(parquetFilePath.getParent(),
+ fullPrefix + parquetFilePath.getName() + KEY_MATERIAL_FILE_SUFFFIX);
+ }
+
+ @Override
+ public void addKeyMaterial(String keyIDInFile, String keyMaterial) throws
ParquetCryptoRuntimeException {
+if (null == keyMaterialMap) {
+ keyMaterialMap = new HashMap<>();
+}
+keyMaterialMap.put(keyIDInFile, keyMaterial);
+ }
+
+ @Override
+ public String getKeyMaterial(String keyIDInFile) throws
ParquetCryptoRuntimeException {
+if (null == keyMaterialMap) {
+ loadKeyMaterialMap();
+}
+return keyMaterialMap.get(keyIDInFile);
+ }
+
+ private void loadKeyMaterialMap() {
+try (FSDataInputStream keyMaterialStream =
hadoopFileSystem.open(keyMaterialFile)) {
+ JsonNode keyMaterialJson = objectMapper.readTree(keyMaterialStream);
+ keyMaterialMap = objectMapper.readValue(keyMaterialJson,
+new TypeReference>() { });
+} catch (IOException e) {
+ throw new ParquetCryptoRuntimeException("Failed to get key material from
" + keyMaterialFile, e);
+}
+ }
+
+ @Override
+ public void saveMaterial() throws ParquetCryptoRuntimeException {
+// TODO needed? Path qualifiedPath =
parquetFilePath.makeQualified(hadoopFileSystem);
+try (FSDataOutputStream keyMaterialStream =
hadoopFileSystem.create(keyMaterialFile)) {
+ objectMapper.writeValue(keyMaterialStream, keyMaterialMap);
+} catch (IOException e) {
+ throw new ParquetCryptoRuntimeException("Failed to save key material in
" + keyMaterialFile, e);
+}
+ }
+
+ @Override
+ public Set getKeyIDSet() throws ParquetCryptoRuntimeException {
+if (null == keyMaterialMap) {
+ loadKeyMaterialMap();
+}
+
+return keyMaterialMap.keySet();
Review comment:
need to walk back this one. We process the footer key entry first, and
then remove it from the set/map to loop on the columns only.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Encryption key management tools
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135456#comment-17135456
]
ASF GitHub Bot commented on PARQUET-1373:
-
ggershinsky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r439953778
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/KeyToolkit.java
##
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+
+package org.apache.parquet.crypto.keytools;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.crypto.AesGcmDecryptor;
+import org.apache.parquet.crypto.AesGcmEncryptor;
+import org.apache.parquet.crypto.AesMode;
+import org.apache.parquet.crypto.KeyAccessDeniedException;
+import org.apache.parquet.crypto.ModuleCipherFactory;
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.apache.parquet.hadoop.BadConfigurationException;
+import org.apache.parquet.hadoop.util.ConfigurationUtil;
+import org.apache.parquet.hadoop.util.HiddenFileFilter;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+
+public class KeyToolkit {
+
+ public static final String KMS_CLIENT_CLASS_PROPERTY_NAME =
"encryption.kms.client.class";
+ public static final String KMS_INSTANCE_ID_PROPERTY_NAME =
"encryption.kms.instance.id";
+ public static final String DOUBLE_WRAPPING_PROPERTY_NAME =
"encryption.double.wrapping";
+ public static final String KEY_ACCESS_TOKEN_PROPERTY_NAME =
"encryption.key.access.token";
+ public static final String TOKEN_LIFETIME_PROPERTY_NAME =
"encryption.key.access.token.lifetime";
+ public static final String KMS_INSTANCE_URL_PROPERTY_NAME =
"encryption.kms.instance.url";
+ public static final String WRAP_LOCALLY_PROPERTY_NAME =
"encryption.wrap.locally";
+ public static final String KEY_MATERIAL_INTERNAL_PROPERTY_NAME =
"encryption.key.material.internal.storage";
Review comment:
Every property name in README.md starts with `"parquet."`. We'll do the
same for the encryption properties.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Encryption key management tools
>
>
> Key: PARQUET-1373
> URL: https://issues.apache.org/jira/browse/PARQUET-1373
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-mr
>Reporter: Gidon Gershinsky
>Assignee: Gidon Gershinsky
>Priority: Major
>
> Parquet Modular Encryption
> ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides
> an API that accepts keys, arbitrary key metadata and key retrieval callbacks
> - which allows to implement basically any key management policy on top of it.
> This Jira will add tools that implement a set of best practice elements for
> key management. This is not an end-to-end key management, but rather a set of
> components that might simplify design and development of an end-to-end
> solution.
> This tool set is one of many possible. There is no goal to create a single or
> “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory
> interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817])
> that allows to plug in different implementations of encryption key management.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135073#comment-17135073
]
ASF GitHub Bot commented on PARQUET-1373:
-
ggershinsky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r439801999
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/samples/VaultClient.java
##
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.crypto.keytools.samples;
+
+import okhttp3.MediaType;
+import okhttp3.OkHttpClient;
+import okhttp3.Request;
+import okhttp3.RequestBody;
+import okhttp3.Response;
+
+import org.apache.parquet.crypto.KeyAccessDeniedException;
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.apache.parquet.crypto.keytools.KmsClient;
+import org.apache.parquet.crypto.keytools.RemoteKmsClient;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+
+public class VaultClient extends RemoteKmsClient {
Review comment:
re samples - we have two classes there, `InMemoryKMS` and `VaultClient`.
`InMemoryKMS` is leveraged in the unitest for the Key Tools feature.
So indeed it should make sense to move it to the tests.
As for the `VaultClient` - one option is to move it also to the tests
location, commenting out the HTTP calls so the `okhttp3` dependency can be
removed from the pom. The comments will explain the code and how to build it if
needed. The other option is to remove `VaultClient` altogether from
apache/parquet-mr (I can keep it eg in my fork, as a pointer if anybody's
interested to see a sample). What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Encryption key management tools
>
>
> Key: PARQUET-1373
> URL: https://issues.apache.org/jira/browse/PARQUET-1373
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-mr
>Reporter: Gidon Gershinsky
>Assignee: Gidon Gershinsky
>Priority: Major
>
> Parquet Modular Encryption
> ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides
> an API that accepts keys, arbitrary key metadata and key retrieval callbacks
> - which allows to implement basically any key management policy on top of it.
> This Jira will add tools that implement a set of best practice elements for
> key management. This is not an end-to-end key management, but rather a set of
> components that might simplify design and development of an end-to-end
> solution.
> This tool set is one of many possible. There is no goal to create a single or
> “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory
> interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817])
> that allows to plug in different implementations of encryption key management.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135061#comment-17135061
]
ASF GitHub Bot commented on PARQUET-1373:
-
ggershinsky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r439800189
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/FileKeyWrapper.java
##
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+
+package org.apache.parquet.crypto.keytools;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.security.SecureRandom;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.apache.parquet.crypto.keytools.KeyToolkit.KeyEncryptionKey;
+import org.codehaus.jackson.map.ObjectMapper;
+
+public class FileKeyWrapper {
+
+ public static final int KEK_LENGTH = 16;
+ public static final int KEK_ID_LENGTH = 16;
+
+ // For every token: a map of MEK_ID to (KEK ID and KEK)
+ private static final ConcurrentMap>> KEKMapPerToken =
+ new ConcurrentHashMap<>(KeyToolkit.INITIAL_PER_TOKEN_CACHE_SIZE);
+ private static volatile long lastKekCacheCleanupTimestamp =
System.currentTimeMillis() + 60l * 1000; // grace period of 1 minute;
+
+ //A map of MEK_ID to (KEK ID and KEK) - for the current token
+ private final ConcurrentMap KEKPerMasterKeyID;
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private final long cacheEntryLifetime;
+
+ private final KmsClient kmsClient;
+ private final String kmsInstanceID;
+ private final String kmsInstanceURL;
+ private final FileKeyMaterialStore keyMaterialStore;
+ private final Configuration hadoopConfiguration;
+ private final SecureRandom random;
+ private final boolean doubleWrapping;
+
+ private short keyCounter;
+ private String accessToken;
+
+ public FileKeyWrapper(Configuration configuration, FileKeyMaterialStore
keyMaterialStore) {
+this.hadoopConfiguration = configuration;
+
+cacheEntryLifetime = 1000l *
hadoopConfiguration.getLong(KeyToolkit.TOKEN_LIFETIME_PROPERTY_NAME,
+KeyToolkit.DEFAULT_CACHE_ENTRY_LIFETIME);
+
+kmsInstanceID =
hadoopConfiguration.getTrimmed(KeyToolkit.KMS_INSTANCE_ID_PROPERTY_NAME,
+KmsClient.DEFAULT_KMS_INSTANCE_ID);
+
+doubleWrapping =
hadoopConfiguration.getBoolean(KeyToolkit.DOUBLE_WRAPPING_PROPERTY_NAME, true);
+accessToken =
hadoopConfiguration.getTrimmed(KeyToolkit.KEY_ACCESS_TOKEN_PROPERTY_NAME,
KmsClient.DEFAULT_ACCESS_TOKEN);
+
+kmsClient = KeyToolkit.getKmsClient(kmsInstanceID, configuration,
accessToken, cacheEntryLifetime);
+
+kmsInstanceURL =
hadoopConfiguration.getTrimmed(KeyToolkit.KMS_INSTANCE_URL_PROPERTY_NAME,
+RemoteKmsClient.DEFAULT_KMS_INSTANCE_URL);
+
+this.keyMaterialStore = keyMaterialStore;
+
+random = new SecureRandom();
+keyCounter = 0;
+
+// Check caches upon each file writing (clean once in cacheEntryLifetime)
+KeyToolkit.checkKmsCacheForExpiredTokens(cacheEntryLifetime);
+if (doubleWrapping) {
+ checkKekCacheForExpiredTokens();
+
+ ExpiringCacheEntry>
KEKCacheEntry = KEKMapPerToken.get(accessToken);
+ if ((null == KEKCacheEntry) || KEKCacheEntry.isExpired()) {
+synchronized (KEKMapPerToken) {
+ KEKCacheEntry = KEKMapPerToken.get(accessToken);
+ if ((null == KEKCacheEntry) || KEKCacheEntry.isExpired()) {
+KEKCacheEntry = new ExpiringCacheEntry<>(new
ConcurrentHashMap(), cacheEntryLifetime);
+KEKMapPerToken.put(accessToken, KEKCacheEntry);
+ }
+}
+ }
+ KEKPerMasterKeyID = KEKCacheEntry.getCachedItem();
+} else {
+ KEKPerMasterKeyID = null;
+}
+ }
+
+ public byte[] getEncryptionKeyMetadata(byte[] dataKey, String masterKeyID,
boolean isFooterKey) {
+return getEncryptionKeyMetadata(dataKey,
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135060#comment-17135060 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky commented on pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#issuecomment-643729252 > There are a couple of concurrent code parts. We shall test somehow that the code is thread-safe. I know, it is not easy but if you execute the related code paths with several parallel threads (e.g. encrypting/decrypting several files in multiple threads) we might find some issues. we'll make the PR test to run multiple threads, writing a number of files in parallel (same for reading). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135054#comment-17135054
]
ASF GitHub Bot commented on PARQUET-1373:
-
ggershinsky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r439797162
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/PropertiesDrivenCryptoFactory.java
##
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.crypto.keytools;
+
+import java.io.IOException;
+import java.security.SecureRandom;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.crypto.ColumnEncryptionProperties;
+import org.apache.parquet.crypto.DecryptionKeyRetriever;
+import org.apache.parquet.crypto.DecryptionPropertiesFactory;
+import org.apache.parquet.crypto.EncryptionPropertiesFactory;
+import org.apache.parquet.crypto.FileDecryptionProperties;
+import org.apache.parquet.crypto.FileEncryptionProperties;
+import org.apache.parquet.crypto.ParquetCipher;
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.apache.parquet.hadoop.api.WriteSupport.WriteContext;
+import org.apache.parquet.hadoop.metadata.ColumnPath;
+
+import static org.apache.parquet.crypto.keytools.KeyToolkit.stringIsEmpty;
+
+public class PropertiesDrivenCryptoFactory implements
EncryptionPropertiesFactory, DecryptionPropertiesFactory {
+
+ public static final String COLUMN_KEYS_PROPERTY_NAME =
"encryption.column.keys";
+ public static final String FOOTER_KEY_PROPERTY_NAME =
"encryption.footer.key";
+ public static final String ENCRYPTION_ALGORITHM_PROPERTY_NAME =
"encryption.algorithm";
+ public static final String PLAINTEXT_FOOTER_PROPERTY_NAME =
"encryption.plaintext.footer";
+
+ public static final int DEK_LENGTH = 16;
+
+ private static final SecureRandom random = new SecureRandom();
+
+ @Override
+ public FileEncryptionProperties getFileEncryptionProperties(Configuration
fileHadoopConfig, Path tempFilePath,
+ WriteContext fileWriteContext) throws ParquetCryptoRuntimeException {
+
+String footerKeyId =
fileHadoopConfig.getTrimmed(FOOTER_KEY_PROPERTY_NAME);
+String columnKeysStr =
fileHadoopConfig.getTrimmed(COLUMN_KEYS_PROPERTY_NAME);
+
+// File shouldn't be encrypted
+if (stringIsEmpty(footerKeyId) && stringIsEmpty(columnKeysStr)) {
+ return null;
+}
+
+if (stringIsEmpty(footerKeyId)) {
+ throw new ParquetCryptoRuntimeException("Undefined footer key");
+}
+
+FileKeyMaterialStore keyMaterialStore = null;
+boolean keyMaterialInternalStorage =
fileHadoopConfig.getBoolean(KeyToolkit.KEY_MATERIAL_INTERNAL_PROPERTY_NAME,
true);
Review comment:
In this particular case, the properties is used twice is two independent
paths - file write path (get encryption properties), and file read path (get
decryption properties). But we can use a const (final bool) for its default
value. And will check other properties.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Encryption key management tools
>
>
> Key: PARQUET-1373
> URL: https://issues.apache.org/jira/browse/PARQUET-1373
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-mr
>Reporter: Gidon Gershinsky
>Assignee: Gidon Gershinsky
>Priority: Major
>
> Parquet Modular Encryption
> ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides
> an API that accepts keys, arbitrary key metadata and key retrieval callbacks
> - which allows to implement basically any key management policy on top of it.
> This Jira will add tools that
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135051#comment-17135051
]
ASF GitHub Bot commented on PARQUET-1373:
-
ggershinsky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r439795858
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/HadoopFSKeyMaterialStore.java
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.crypto.keytools;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.codehaus.jackson.JsonNode;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.codehaus.jackson.type.TypeReference;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+public class HadoopFSKeyMaterialStore implements FileKeyMaterialStore {
+
+ public final static String KEY_MATERIAL_FILE_PREFIX = "_KEY_MATERIAL_FOR_";
+ public static final String TEMP_FILE_PREFIX = "_TMP";
+ public final static String KEY_MATERIAL_FILE_SUFFFIX = ".json";
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private FileSystem hadoopFileSystem;
+ private Map keyMaterialMap;
+ private Path keyMaterialFile;
+
+ HadoopFSKeyMaterialStore(FileSystem hadoopFileSystem) {
+this.hadoopFileSystem = hadoopFileSystem;
+ }
+
+ @Override
+ public void initialize(Path parquetFilePath, Configuration hadoopConfig,
boolean tempStore) {
+String fullPrefix = (tempStore? TEMP_FILE_PREFIX : "");
+fullPrefix += KEY_MATERIAL_FILE_PREFIX;
+keyMaterialFile = new Path(parquetFilePath.getParent(),
+ fullPrefix + parquetFilePath.getName() + KEY_MATERIAL_FILE_SUFFFIX);
+ }
+
+ @Override
+ public void addKeyMaterial(String keyIDInFile, String keyMaterial) throws
ParquetCryptoRuntimeException {
+if (null == keyMaterialMap) {
+ keyMaterialMap = new HashMap<>();
+}
+keyMaterialMap.put(keyIDInFile, keyMaterial);
+ }
+
+ @Override
+ public String getKeyMaterial(String keyIDInFile) throws
ParquetCryptoRuntimeException {
+if (null == keyMaterialMap) {
+ loadKeyMaterialMap();
+}
+return keyMaterialMap.get(keyIDInFile);
+ }
+
+ private void loadKeyMaterialMap() {
+try (FSDataInputStream keyMaterialStream =
hadoopFileSystem.open(keyMaterialFile)) {
+ JsonNode keyMaterialJson = objectMapper.readTree(keyMaterialStream);
+ keyMaterialMap = objectMapper.readValue(keyMaterialJson,
+new TypeReference>() { });
+} catch (IOException e) {
+ throw new ParquetCryptoRuntimeException("Failed to get key material from
" + keyMaterialFile, e);
+}
+ }
+
+ @Override
+ public void saveMaterial() throws ParquetCryptoRuntimeException {
+// TODO needed? Path qualifiedPath =
parquetFilePath.makeQualified(hadoopFileSystem);
Review comment:
will remove the comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Encryption key management tools
>
>
> Key: PARQUET-1373
> URL: https://issues.apache.org/jira/browse/PARQUET-1373
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-mr
>Reporter: Gidon Gershinsky
>Assignee: Gidon Gershinsky
>Priority: Major
>
> Parquet Modular Encryption
> ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides
> an API that accepts keys, arbitrary key metadata and key retrieval callbacks
> - which allows to implement basically any key management
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135046#comment-17135046
]
ASF GitHub Bot commented on PARQUET-1373:
-
ggershinsky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r439794717
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/FileKeyUnwrapper.java
##
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.parquet.crypto.keytools;
+
+import java.io.IOException;
+import java.io.StringReader;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.crypto.DecryptionKeyRetriever;
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.apache.parquet.crypto.keytools.KeyToolkit.KeyWithMasterID;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.codehaus.jackson.type.TypeReference;
+
+import static org.apache.parquet.crypto.keytools.KeyToolkit.stringIsEmpty;
+
+public class FileKeyUnwrapper implements DecryptionKeyRetriever {
+ // For every token: a map of KEK_ID to KEK bytes
+ private static final ConcurrentMap>> KEKMapPerToken = new
ConcurrentHashMap<>();
Review comment:
The KEK purpose is to minimize the Parquet-KMS interaction. Without it,
each thread would have to go to the KMS to unwrap a key. By making the cache
static, we provide the KEK to all threads (with the same token), if one of them
has already unwrapped it. Will add this to the comments.
We'll check the use of singletons for this. The three maps/caches are not
identical, but we'll make an effort to unify the cache handling.
ConcurrentMap has a segment synchronization for write operations, and allows
for synchronization-free read operations; this makes it faster than HasMap with
synchronized methods.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Encryption key management tools
>
>
> Key: PARQUET-1373
> URL: https://issues.apache.org/jira/browse/PARQUET-1373
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-mr
>Reporter: Gidon Gershinsky
>Assignee: Gidon Gershinsky
>Priority: Major
>
> Parquet Modular Encryption
> ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides
> an API that accepts keys, arbitrary key metadata and key retrieval callbacks
> - which allows to implement basically any key management policy on top of it.
> This Jira will add tools that implement a set of best practice elements for
> key management. This is not an end-to-end key management, but rather a set of
> components that might simplify design and development of an end-to-end
> solution.
> This tool set is one of many possible. There is no goal to create a single or
> “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory
> interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817])
> that allows to plug in different implementations of encryption key management.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135039#comment-17135039
]
ASF GitHub Bot commented on PARQUET-1373:
-
ggershinsky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r439793036
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/KeyToolkit.java
##
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+
+package org.apache.parquet.crypto.keytools;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.crypto.AesGcmDecryptor;
+import org.apache.parquet.crypto.AesGcmEncryptor;
+import org.apache.parquet.crypto.AesMode;
+import org.apache.parquet.crypto.KeyAccessDeniedException;
+import org.apache.parquet.crypto.ModuleCipherFactory;
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.apache.parquet.hadoop.BadConfigurationException;
+import org.apache.parquet.hadoop.util.ConfigurationUtil;
+import org.apache.parquet.hadoop.util.HiddenFileFilter;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+
+public class KeyToolkit {
+
+ public static final String KMS_CLIENT_CLASS_PROPERTY_NAME =
"encryption.kms.client.class";
+ public static final String KMS_INSTANCE_ID_PROPERTY_NAME =
"encryption.kms.instance.id";
+ public static final String DOUBLE_WRAPPING_PROPERTY_NAME =
"encryption.double.wrapping";
+ public static final String KEY_ACCESS_TOKEN_PROPERTY_NAME =
"encryption.key.access.token";
+ public static final String TOKEN_LIFETIME_PROPERTY_NAME =
"encryption.key.access.token.lifetime";
+ public static final String KMS_INSTANCE_URL_PROPERTY_NAME =
"encryption.kms.instance.url";
+ public static final String WRAP_LOCALLY_PROPERTY_NAME =
"encryption.wrap.locally";
+ public static final String KEY_MATERIAL_INTERNAL_PROPERTY_NAME =
"encryption.key.material.internal.storage";
+
+ public static final String KEY_MATERIAL_TYPE_FIELD = "keyMaterialType";
+ public static final String KEY_MATERIAL_TYPE = "PKMT1";
+ public static final String KEY_MATERIAL_INTERNAL_STORAGE_FIELD =
"internalStorage";
+ public static final String KEY_REFERENCE_FIELD = "keyReference";
+ public static final String DOUBLE_WRAPPING_FIELD = "doubleWrapping";
+
+ public static final String KMS_INSTANCE_ID_FIELD = "kmsInstanceID";
+ public static final String KMS_INSTANCE_URL_FIELD = "kmsInstanceURL";
+
+ public static final String MASTER_KEY_ID_FIELD = "masterKeyID";
+ public static final String WRAPPED_DEK_FIELD = "wrappedDEK";
+ public static final String KEK_ID_FIELD = "keyEncryptionKeyID";
+ public static final String WRAPPED_KEK_FIELD = "wrappedKEK";
+
+ public static final String FOOTER_KEY_ID_IN_FILE = "kf";
+ public static final String KEY_ID_IN_FILE_PREFIX = "k";
Review comment:
changed to `"footerKey"`, and `COLUMN_KEY_ID_IN_FILE_PREFIX =
"columnKey"`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Encryption key management tools
>
>
> Key: PARQUET-1373
> URL: https://issues.apache.org/jira/browse/PARQUET-1373
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-mr
>Reporter: Gidon Gershinsky
>Assignee: Gidon Gershinsky
>Priority: Major
>
> Parquet Modular Encryption
> ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides
> an API that accepts keys, arbitrary key metadata and key retrieval
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[
https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1715#comment-1715
]
ASF GitHub Bot commented on PARQUET-1373:
-
gszadovszky commented on a change in pull request #615:
URL: https://github.com/apache/parquet-mr/pull/615#discussion_r435236094
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/FileKeyUnwrapper.java
##
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.parquet.crypto.keytools;
+
+import java.io.IOException;
+import java.io.StringReader;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.crypto.DecryptionKeyRetriever;
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.apache.parquet.crypto.keytools.KeyToolkit.KeyWithMasterID;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.codehaus.jackson.type.TypeReference;
+
+import static org.apache.parquet.crypto.keytools.KeyToolkit.stringIsEmpty;
+
+public class FileKeyUnwrapper implements DecryptionKeyRetriever {
+ // For every token: a map of KEK_ID to KEK bytes
+ private static final ConcurrentMap>> KEKMapPerToken = new
ConcurrentHashMap<>();
Review comment:
As per the java naming conventions for constants (`static final`):
```suggestion
private static final ConcurrentMap>> KEK_MAP_PER_TOKEN = new
ConcurrentHashMap<>();
```
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/FileKeyUnwrapper.java
##
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.parquet.crypto.keytools;
+
+import java.io.IOException;
+import java.io.StringReader;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.crypto.DecryptionKeyRetriever;
+import org.apache.parquet.crypto.ParquetCryptoRuntimeException;
+import org.apache.parquet.crypto.keytools.KeyToolkit.KeyWithMasterID;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.codehaus.jackson.type.TypeReference;
+
+import static org.apache.parquet.crypto.keytools.KeyToolkit.stringIsEmpty;
+
+public class FileKeyUnwrapper implements DecryptionKeyRetriever {
+ // For every token: a map of KEK_ID to KEK bytes
+ private static final ConcurrentMap>> KEKMapPerToken = new
ConcurrentHashMap<>();
+ private volatile static long lastKekCacheCleanupTimestamp =
System.currentTimeMillis() + 60l * 1000; // grace period of 1 minute
+ //A map of KEK_ID to KEK - for the current token
+ private final ConcurrentMap KEKPerKekID;
Review comment:
nit: should start with lower case. Though, the naming conventions for
acronyms are not clear I would vote on the following.
```suggestion
private final ConcurrentMap kekPerKekID;
```
##
File path:
parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/FileKeyWrapper.java
##
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125745#comment-17125745 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky commented on pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#issuecomment-638737727 > @ggershinsky, is it still in draft state? @gszadovszky The pr is completed, ready for a review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125659#comment-17125659 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky commented on pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#issuecomment-638677887 the implementation is complete. with the recent addition of a unitest, the draft state can be switched off. I'll rebase it first, to make sure the Travis checks pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125604#comment-17125604 ] ASF GitHub Bot commented on PARQUET-1373: - gszadovszky commented on pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#issuecomment-638646464 @ggershinsky, is it still in draft state? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
