Mike Dias created HADOOP-17855:
----------------------------------
Summary: Allow SSE configurations per object path
Key: HADOOP-17855
URL: https://issues.apache.org/jira/browse/HADOOP-17855
Project: Hadoop Common
Issue Type: New Feature
Components: fs/s3
Reporter: Mike Dias
Currently, we can map the SSE configurations at bucket level only:
{code:java}
<property>
<name>fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm</name>
<value>SSE-KMS</value>
</property>
<property>
<name>fs.s3a.bucket.ireland-dev.server-side-encryption.key</name>
<value>arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01</value>
</property>
{code}
But sometimes we want to encrypt data in different paths with different keys
within the same bucket. For example, a partitioned table might benefit from
encrypting each partition with a different key when the partition represents a
customer or a country.
[S3 already can encrypt using different keys/configurations at the object
level|https://aws.amazon.com/premiumsupport/knowledge-center/s3-encrypt-specific-folder/],
so what we need to do on Hadoop is to provide a way to map which key to use.
One idea could be mapping them in the XML config:
{code:java}
<property>
<name>fs.s3a.server-side-encryption.paths</name>
<value>s3://bucket/my_table/country=ireland,s3://bucket/my_table/country=uk,
s3://bucket/my_table/country=germany</value>
</property>
<property>
<name>fs.s3a.server-side-encryption.path-keys</name>
<value>arn:aws:kms:eu-west-1:90ireland09:key/ireland-key,arn:aws:kms:eu-west-1:980uk0993c:key/uk-key,arn:aws:kms:eu-west-1:98germany089:key/germany-key</value>
</property>
{code}
Or potentially fetch the mappings from the filesystem:
{code:java}
<property>
<name>fs.s3a.server-side-encryption.mappings</name>
<value>s3://bucket/configs/encryption_mappings.json</value>
</property> {code}
where encryption_mappings.json could be something like this:
{code:java}
{
"path": "s3://bucket/customer_table/customerId=abc123",
"algorithm": "SSE-KMS",
"key": "arn:aws:kms:eu-west-1:933993746:key/abc123-key"
}
...
{
"path": "s3://bucket/customer_table/customerId=xyx987",
"algorithm": "SSE-KMS",
"key": "arn:aws:kms:eu-west-1:933993746:key/xyx987-key"
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
