bharatviswa504 opened a new pull request #2467: URL: https://github.com/apache/ozone/pull/2467
## What changes were proposed in this pull request? Add support to put/get/mpu from encrypted bucket from S3G. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-5501 ## How was this patch tested? ``` Manually tested the fix on secure KMS Cluster. **S3Gatway** <property><name>ozone.s3g.kerberos.keytab.file</name><value>/tmp/ozone.keytab</value></property> <property><name>ozone.s3g.kerberos.principal.key</name><value>s3g/[email protected]</value></property> **Ranger KMS** <property><name>hadoop.kms.proxyuser.s3g.users</name><value>*</value></property <property><name>hadoop.kms.proxyuser.s3g.hosts</name><value>*</value></property><property><name>hadoop.kms.proxyuser.s3g.groups</name><value>*</value></property> Configure credentials [root@nightly71x-4 ~]# kinit -kt /var/run/cloudera-scm-agent/process/71-hbase-REGIONSERVER/hbase.keytab hbase/[email protected] [root@nightly71x-4 ~]# ozone s3 getsecret --om-service-id=ozone1 awsAccessKey=hbase/[email protected] awsSecret=28805795d4118e1aa4acbd70c7f6915384a30af7ac55f942ccc260ba999bcbba [root@nightly71x-4 ~]# aws configure AWS Access Key ID [****************SITE]: hbase/[email protected] AWS Secret Access Key [****************cbba]: 28805795d4118e1aa4acbd70c7f6915384a30af7ac55f942ccc260ba999bcbba Default region name [None]: Default output format [None]: Test Create encryption key hadoop key create key1 **Create enc buck** [root@nightly71x-4 ~]# ozone sh bucket create /s3v/s3encbuck --bucketkey key1 21/07/26 11:23:06 INFO rpc.RpcClient: Creating Bucket: s3v/s3encbuck, with Versioning false and Storage Type set to DISK and Encryption set to true **S3 Test** [root@nightly71x-4 ~]# alias s3api='aws s3api --endpoint https://nightly71x-4.nightly71x.root.hwx.site:9879 --no-verify-ssl' [root@nightly71x-4 ~]# s3api put-object --bucket s3encbuck --key key1 --body /etc/hadoop/conf/ozone-site.xml /usr/lib/fence-agents/bundled/botocore/vendored/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) [root@nightly71x-4 ~]# s3api get-object --bucket s3encbuck --key key1 /tmp/key1 /usr/lib/fence-agents/bundled/botocore/vendored/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) { "AcceptRanges": "bytes", "ContentType": "application/octet-stream", "LastModified": "Mon, 26 Jul 2021 11:24:21 GMT", "ContentLength": 3785, "Expires": "Mon, 26 Jul 2021 11:25:01 GMT", "CacheControl": "no-cache", "Metadata": {} } [root@nightly71x-4 ~]# diff /tmp/key1 /etc/hadoop/conf/ozone-site.xml [root@nightly71x-4 ~]# s3api list-objects --bucket s3encbuck /usr/lib/fence-agents/bundled/botocore/vendored/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) { "Contents": [ { "LastModified": "2021-07-26T11:24:21.458Z", "ETag": "2021-07-26T11:24:21.458Z", "StorageClass": "STANDARD", "Key": "key1", "Size": 3785 } ] } ``` **Shell Test** ``` [root@nightly71x-4 ~]# ozone sh key put /s3v/s3encbuck/key1 /etc/hadoop/conf/ozone-site.xml [root@nightly71x-4 ~]# ozone sh key get /s3v/s3encbuck/key1 /tmp/dkey1 [root@nightly71x-4 ~]# diff /tmp/dkey1 /etc/hadoop/conf/ozone-site.xml ``` Opened a jira https://issues.apache.org/jira/browse/HDDS-5507 to add secure KMS and add tests for put/get/mpu from encrypted bucket using s3g. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
