HDDS-677. Create documentation for s3 gateway to the docs. Contributed by Elek Marton.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/6668c19d Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/6668c19d Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/6668c19d Branch: refs/heads/HDDS-4 Commit: 6668c19dafde530c43ccacb23de11455ae1813b5 Parents: 08bb036 Author: Bharat Viswanadham <bha...@apache.org> Authored: Wed Oct 31 11:48:23 2018 -0700 Committer: Bharat Viswanadham <bha...@apache.org> Committed: Wed Oct 31 11:48:23 2018 -0700 ---------------------------------------------------------------------- hadoop-ozone/docs/content/S3.md | 130 +++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/6668c19d/hadoop-ozone/docs/content/S3.md ---------------------------------------------------------------------- diff --git a/hadoop-ozone/docs/content/S3.md b/hadoop-ozone/docs/content/S3.md new file mode 100644 index 0000000..cfefaf4 --- /dev/null +++ b/hadoop-ozone/docs/content/S3.md @@ -0,0 +1,130 @@ +--- +title: S3 +menu: + main: + parent: Client + weight: 1 +--- + +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools. + +## Getting started + +S3 Gateway is a separated component which provides the S3 compatible. It should be started additional to the regular Ozone components. + +You can start a docker based cluster, including the S3 gateway from the release package. + +Go to the `compose/ozones3` directory, and start the server: + +```bash +docker-compose up -d +``` + +You can access the S3 gateway at `http://localhost:9878` + +## URL Schema + +Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. http://bucketname.host:9878) and the path-style addresses (eg. http://host:9878/bucketname) + +By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your `ozone-site.xml`: + +```xml +<property> + <name>ozone.s3g.domain.name</name> + <value>s3g.internal</value> +</property> +``` + +## Bucket browser + +Bucket's could be browsed from the browser with adding `?browser=true` to the bucket URL. + +For example the content of the 'testbucket' could be checked from the browser using the URL http://localhost:9878/testbucket?browser=true + + +## Implemented REST endpoints + +Operations on S3Gateway service: + +Endpoint | Status | +------------|-------------| +GET service | implemented | + +Operations on Bucket: + +Endpoint | Status | Notes +------------------------------------|-------------|--------------- +GET Bucket (List Objects) Version 2 | implemented | +HEAD Bucket | implemented | +DELETE Bucket | implemented | +PUT Bucket (Create bucket) | implemented | +Delete Multiple Objects (POST) | implemented | + +Operation on Objects: + +Endpoint | Status | Notes +------------------------------------|-----------------|--------------- +PUT Object | implemented | +GET Object | implemented | Range headers are not supported +Multipart Uplad | not implemented | +DELETE Object | implemented | +HEAD Object | implemented | + + +## Security + +Security is not yet implemented, you can *use* any AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY + +Note: Ozone has a notion for 'volumes' which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are maped to Ozone 'volume/bucket' locations (depending from the given authentication information). + +To show the storage location of a S3 bucket, use the `ozone sh bucket path <bucketname>` command. + +``` +aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket=bucket1 + +ozone sh bucket path bucket1 +Volume name for S3Bucket is : s3thisisakey +Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey +``` + +## Clients + +### AWS Cli + +`aws` CLI could be used with specifying the custom REST endpoint. + +``` +aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest +``` + +Or + +``` +aws s3 ls --endpoint http://localhost:9878 s3://buckettest +``` + +### S3 Fuse driver (goofys) + +Goofys is a S3 FUSE driver. It could be used to any mount any Ozone bucket as posix file system: + + +``` +goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1 +``` --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
