[
https://issues.apache.org/jira/browse/HADOOP-17855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-17855:
------------------------------------
Parent: HADOOP-17566
Issue Type: Sub-task (was: New Feature)
> S3A: Allow SSE configurations per object path
> ---------------------------------------------
>
> Key: HADOOP-17855
> URL: https://issues.apache.org/jira/browse/HADOOP-17855
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Mike Dias
> Priority: Major
>
> Currently, we can map the SSE configurations at bucket level only:
> {code:java}
> <property>
> <name>fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm</name>
> <value>SSE-KMS</value>
> </property>
> <property>
> <name>fs.s3a.bucket.ireland-dev.server-side-encryption.key</name>
>
> <value>arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01</value>
> </property>
> {code}
> But sometimes we want to encrypt data in different paths with different keys
> within the same bucket. For example, a partitioned table might benefit from
> encrypting each partition with a different key when the partition represents
> a customer or a country.
> [S3 already can encrypt using different keys/configurations at the object
> level|https://aws.amazon.com/premiumsupport/knowledge-center/s3-encrypt-specific-folder/],
> so what we need to do on Hadoop is to provide a way to map which key to use.
> One idea could be mapping them in the XML config:
>
> {code:java}
> <property>
> <name>fs.s3a.server-side-encryption.paths</name>
>
> <value>s3://bucket/my_table/country=ireland,s3://bucket/my_table/country=uk,
> s3://bucket/my_table/country=germany</value>
> </property>
> <property>
> <name>fs.s3a.server-side-encryption.path-keys</name>
>
> <value>arn:aws:kms:eu-west-1:90ireland09:key/ireland-key,arn:aws:kms:eu-west-1:980uk0993c:key/uk-key,arn:aws:kms:eu-west-1:98germany089:key/germany-key</value>
> </property>
> {code}
> Or potentially fetch the mappings from the filesystem:
>
> {code:java}
> <property>
> <name>fs.s3a.server-side-encryption.mappings</name>
> <value>s3://bucket/configs/encryption_mappings.json</value>
> </property> {code}
> where encryption_mappings.json could be something like this:
>
> {code:java}
> {
> "path": "s3://bucket/customer_table/customerId=abc123",
> "algorithm": "SSE-KMS",
> "key": "arn:aws:kms:eu-west-1:933993746:key/abc123-key"
> }
> ...
> {
> "path": "s3://bucket/customer_table/customerId=xyx987",
> "algorithm": "SSE-KMS",
> "key": "arn:aws:kms:eu-west-1:933993746:key/xyx987-key"
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
