[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149592#comment-17149592 ] Jason Brugger commented on PARQUET-1373: unsubscribe On Wed, Jul 1, 2020 at 9:31 AM ASF GitHub Bot (Jira) > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149477#comment-17149477 ] ASF GitHub Bot commented on PARQUET-1373: - gszadovszky closed pull request #615: URL: https://github.com/apache/parquet-mr/pull/615 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149476#comment-17149476 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky opened a new pull request #615: URL: https://github.com/apache/parquet-mr/pull/615 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [parquet-mr] gszadovszky closed pull request #615: PARQUET-1373: Encryption key tools
gszadovszky closed pull request #615: URL: https://github.com/apache/parquet-mr/pull/615 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [parquet-mr] ggershinsky commented on a change in pull request #615: PARQUET-1373: Encryption key tools
ggershinsky commented on a change in pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#discussion_r448263545 ## File path: parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/KeyMaterial.java ## @@ -28,14 +28,35 @@ import org.codehaus.jackson.map.ObjectMapper; import org.codehaus.jackson.type.TypeReference; +/** + * KeyMaterial class represents the "key material", keeping the information that allows readers to recover an encryption key (see + * description of the KeyMetadata class). The keytools package (PARQUET-1373) implements the "envelope encryption" pattern, in a + * "single wrapping" or "double wrapping" mode. In the single wrapping mode, the key material is generated by encrypting the + * "data encryption key" (DEK) by a "master key". In the double wrapping mode, the key material is generated by encrypting the DEK + * by a "key encryption key" (KEK), that in turn is encrypted by a "master key". + * + * Key material is kept in a flat json object, with the following fields: + * 1. "keyMaterialType" - a String, with the type of key material. In the current version, only one value is allowed - "PKMT1" (stands + * for "parquet key management tools, version 1"). For external key material storage, this field is written in both "key metadata" and + * "key material" jsons. For internal key material storage, this field is written only once in the common json. + * 2. "isFooterKey" - a boolean. If true, means that the material belongs to a file footer key, and keeps additional information (such as Review comment: a good point, thanks for noticing this. will be fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149252#comment-17149252 ] ASF GitHub Bot commented on PARQUET-1373: - gszadovszky commented on a change in pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#discussion_r448244268 ## File path: parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/KeyMaterial.java ## @@ -28,14 +28,35 @@ import org.codehaus.jackson.map.ObjectMapper; import org.codehaus.jackson.type.TypeReference; +/** + * KeyMaterial class represents the "key material", keeping the information that allows readers to recover an encryption key (see + * description of the KeyMetadata class). The keytools package (PARQUET-1373) implements the "envelope encryption" pattern, in a + * "single wrapping" or "double wrapping" mode. In the single wrapping mode, the key material is generated by encrypting the + * "data encryption key" (DEK) by a "master key". In the double wrapping mode, the key material is generated by encrypting the DEK + * by a "key encryption key" (KEK), that in turn is encrypted by a "master key". + * + * Key material is kept in a flat json object, with the following fields: + * 1. "keyMaterialType" - a String, with the type of key material. In the current version, only one value is allowed - "PKMT1" (stands + * for "parquet key management tools, version 1"). For external key material storage, this field is written in both "key metadata" and + * "key material" jsons. For internal key material storage, this field is written only once in the common json. + * 2. "isFooterKey" - a boolean. If true, means that the material belongs to a file footer key, and keeps additional information (such as Review comment: This (and the other _"boolean"_ members) are not `boolean` values in the as per the json spec. (You are writing `"true"` and `"false"` instead of `true` and `false`.) You may either mention it in the description or try to use a `Map` and put `String` or `Boolean` values. I think, `jackson` will write proper `boolean` values in this case. I would vote on the latter if it works. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [parquet-mr] gszadovszky commented on a change in pull request #615: PARQUET-1373: Encryption key tools
gszadovszky commented on a change in pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#discussion_r448244268 ## File path: parquet-hadoop/src/main/java/org/apache/parquet/crypto/keytools/KeyMaterial.java ## @@ -28,14 +28,35 @@ import org.codehaus.jackson.map.ObjectMapper; import org.codehaus.jackson.type.TypeReference; +/** + * KeyMaterial class represents the "key material", keeping the information that allows readers to recover an encryption key (see + * description of the KeyMetadata class). The keytools package (PARQUET-1373) implements the "envelope encryption" pattern, in a + * "single wrapping" or "double wrapping" mode. In the single wrapping mode, the key material is generated by encrypting the + * "data encryption key" (DEK) by a "master key". In the double wrapping mode, the key material is generated by encrypting the DEK + * by a "key encryption key" (KEK), that in turn is encrypted by a "master key". + * + * Key material is kept in a flat json object, with the following fields: + * 1. "keyMaterialType" - a String, with the type of key material. In the current version, only one value is allowed - "PKMT1" (stands + * for "parquet key management tools, version 1"). For external key material storage, this field is written in both "key metadata" and + * "key material" jsons. For internal key material storage, this field is written only once in the common json. + * 2. "isFooterKey" - a boolean. If true, means that the material belongs to a file footer key, and keeps additional information (such as Review comment: This (and the other _"boolean"_ members) are not `boolean` values in the as per the json spec. (You are writing `"true"` and `"false"` instead of `true` and `false`.) You may either mention it in the description or try to use a `Map` and put `String` or `Boolean` values. I think, `jackson` will write proper `boolean` values in this case. I would vote on the latter if it works. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PARQUET-1373) Encryption key management tools
[ https://issues.apache.org/jira/browse/PARQUET-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149234#comment-17149234 ] ASF GitHub Bot commented on PARQUET-1373: - ggershinsky commented on pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#issuecomment-652301855 @gszadovszky the latest review round is addressed by 0355f65 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Encryption key management tools > > > Key: PARQUET-1373 > URL: https://issues.apache.org/jira/browse/PARQUET-1373 > Project: Parquet > Issue Type: New Feature > Components: parquet-mr >Reporter: Gidon Gershinsky >Assignee: Gidon Gershinsky >Priority: Major > > Parquet Modular Encryption > ([PARQUET-1178|https://issues.apache.org/jira/browse/PARQUET-1178]) provides > an API that accepts keys, arbitrary key metadata and key retrieval callbacks > - which allows to implement basically any key management policy on top of it. > This Jira will add tools that implement a set of best practice elements for > key management. This is not an end-to-end key management, but rather a set of > components that might simplify design and development of an end-to-end > solution. > This tool set is one of many possible. There is no goal to create a single or > “standard” toolkit for Parquet encryption keys. Parquet has a Crypto Factory > interface [(PARQUET-1817|https://issues.apache.org/jira/browse/PARQUET-1817]) > that allows to plug in different implementations of encryption key management. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [parquet-mr] ggershinsky commented on pull request #615: PARQUET-1373: Encryption key tools
ggershinsky commented on pull request #615: URL: https://github.com/apache/parquet-mr/pull/615#issuecomment-652301855 @gszadovszky the latest review round is addressed by 0355f65 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
