adoroszlai commented on a change in pull request #1960: URL: https://github.com/apache/ozone/pull/1960#discussion_r583091674
########## File path: hadoop-hdds/docs/content/recipe/BotoClient.md ########## @@ -0,0 +1,189 @@ +--- +title: Access Ozone object store with Amazon Boto3 client +linktitle: Ozone with Boto3 Client +summary: How to access Ozone object store with Boto3 client? +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +This recipe shows how Ozone object store can be accessed from Boto3 client. Following apis were verified: + + - Create bucket + - List bucket + - Head bucket + - Delete bucket + - Upload file + - Download file + - Delete objects(keys) + - Head object + - Multipart upload + + +## Requirements + +You will need a higher version of Python3 for your Boto3 client as Boto3 installation requirement indicates at here: +https://boto3.amazonaws.com/v1/documentation/api/latest/index.html + +## Obtain resource to Ozone +You may reference Amazon Boto3 documentation regarding the creation of 's3' resources at here: +https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html + + s3 = boto3.resource('s3', + endpoint_url='http://localhost:9878', + aws_access_key_id='testuser/[email protected]', + aws_secret_access_key='c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999' + ) + 'endpoint_url' is pointing to Ozone s3 endpoint. + + +## Obtain client to Ozone via session +You many reference Amazon Boto3 documentation regarding session at here: Review comment: ```suggestion You may reference Amazon Boto3 documentation regarding session at here: ``` ########## File path: hadoop-ozone/dist/src/main/smoketest/botoclient/boto3.robot ########## @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +*** Settings *** +Documentation S3 gateway test with Boto3 Client +Library OperatingSystem +Library String +Library Process +Library BuiltIn +Resource ../commonlib.robot +Resource ../s3/commonawslib.robot +Test Timeout 15 minutes +Suite Setup Setup s3 tests + +*** Variables *** +${ENDPOINT_URL} http://s3g:9878 +${BUCKET} generated + +*** Test Cases *** + +Bobo3 Client Test + ${result} = Execute python3 /opt/hadoop/smoketest/botoclient/boto_client.py + Should contain ${result} Boto3 Client Test PASSED! Review comment: Nit: `Execute` checks the command's return code (expects 0), so no need to store/check `${result}`. ########## File path: hadoop-ozone/dist/src/main/smoketest/botoclient/boto3.robot ########## @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +*** Settings *** +Documentation S3 gateway test with Boto3 Client +Library OperatingSystem +Library String +Library Process +Library BuiltIn +Resource ../commonlib.robot +Resource ../s3/commonawslib.robot +Test Timeout 15 minutes +Suite Setup Setup s3 tests + +*** Variables *** +${ENDPOINT_URL} http://s3g:9878 +${BUCKET} generated + +*** Test Cases *** + +Bobo3 Client Test + ${result} = Execute python3 /opt/hadoop/smoketest/botoclient/boto_client.py Review comment: `${ENDPOINT_URL}` should be passed to the test (and it should use it instead of hard-coded URL). ########## File path: hadoop-ozone/dist/src/main/compose/ozone/test.sh ########## @@ -47,6 +47,8 @@ for bucket in link generated; do execute_robot_test scm -v BUCKET:${bucket} -N s3-${bucket} s3 done +execute_robot_test scm -v BUCKET:link -N s3-link botoclient Review comment: I suggest: * moving `boto3.robot` and `boto_client.py` to `smoketest/s3` * removing this line * updating path of `boto_client.py` in `boto3.robot` * using `${BUCKET}` in `boto_client.py` for testing file upload, list bucket, etc.(most test cases except bucket creation) instead of `bucket1` created by the test With that `boto3` test would get executed for all the environments and parameters where we run the complete `s3` suite of existing `awscli`-based tests (secure/unsecure, HA/non-HA, regular buckets and links). ########## File path: hadoop-hdds/docs/content/recipe/BotoClient.md ########## @@ -0,0 +1,189 @@ +--- +title: Access Ozone object store with Amazon Boto3 client +linktitle: Ozone with Boto3 Client +summary: How to access Ozone object store with Boto3 client? +--- +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +This recipe shows how Ozone object store can be accessed from Boto3 client. Following apis were verified: + + - Create bucket + - List bucket + - Head bucket + - Delete bucket + - Upload file + - Download file + - Delete objects(keys) + - Head object + - Multipart upload + + +## Requirements + +You will need a higher version of Python3 for your Boto3 client as Boto3 installation requirement indicates at here: +https://boto3.amazonaws.com/v1/documentation/api/latest/index.html + +## Obtain resource to Ozone +You may reference Amazon Boto3 documentation regarding the creation of 's3' resources at here: +https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html + + s3 = boto3.resource('s3', + endpoint_url='http://localhost:9878', + aws_access_key_id='testuser/[email protected]', + aws_secret_access_key='c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999' + ) + 'endpoint_url' is pointing to Ozone s3 endpoint. + + +## Obtain client to Ozone via session +You many reference Amazon Boto3 documentation regarding session at here: +https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html + + Create a session + session = boto3.session.Session() + + Obtain s3 client to Ozone via session: + + s3_client = session.client( + service_name='s3', + aws_access_key_id='testuser/[email protected]', + aws_secret_access_key='c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999', + endpoint_url='http://localhost:9878', + ) + 'endpoint_url' is pointing to Ozone s3 endpoint. + + In our code sample below, we're demonstrating the useage of both s3 and s3_client. Review comment: ```suggestion In our code sample below, we're demonstrating the usage of both s3 and s3_client. ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
