jojochuang commented on code in PR #194: URL: https://github.com/apache/ozone-site/pull/194#discussion_r2670580073
########## docs/04-user-guide/01-client-interfaces/03-s3.md: ########## @@ -1,3 +1,189 @@ +--- +sidebar_label: S3 API +--- + # S3 API -**TODO:** File a subtask under [HDDS-9858](https://issues.apache.org/jira/browse/HDDS-9858) and complete this page or section. +Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools. +S3 buckets are stored under the `/s3v` volume. + +## Getting started + +S3 Gateway is a separated component which provides the S3 compatible APIs. It should be started additional to the regular Ozone components. +You can start a Docker based cluster, including the S3 Gateway from the release package. + +Go to the `compose/ozone` directory, and start the server: + +```bash +docker-compose up -d --scale datanode=3 +``` + +You can access the S3 Gateway at `http://localhost:9878` + +## URL Schema + +Ozone S3 Gateway supports both the virtual-host-style URL S3 bucket addresses (eg. http://bucketname.host:9878) and the path-style addresses (eg. http://host:9878/bucketname) +By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your `ozone-site.xml`: + +```xml +<property> + <name>ozone.s3g.domain.name</name> + <value>s3g.internal</value> +</property> +``` + +## Implemented REST Endpoints + +The Ozone S3 Gateway implements a substantial subset of the Amazon S3 REST API. The tables below summarize each API’s support status, its feature, and any relevant notes, including known deviations from AWS S3 behavior. + +### Bucket Operations + +| **API Name** | **Feature** | **Note** | +| --------------------------------------------------------------------------------------------------- | ------------- | ---------- | +| ✅ [ListBuckets](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html) | Lists all buckets owned by the authenticated user. | Returns the full bucket list. | +| ✅ [CreateBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html) | Creates a new bucket. | **Non-compliant behavior:** The default bucket ACL may include extra group permissions instead of being strictly private. Bucket names must adhere to S3 naming conventions. | +| ✅ [HeadBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html) | Checks for the existence of a bucket. | Returns a 200 status if the bucket exists. | +| ✅ [DeleteBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html) | Deletes a bucket. | Bucket must be empty before deletion. | +| ✅ [GetBucketLocation](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html) | Retrieves the location (region) of a bucket. | Typically returns a default region (e.g., `us-east-1`), which may differ from AWS if region-specific responses are expected. | + +### Object Operations + +| **API Name** | **Feature** | **Note** | +| ------------------------------------------------------------------------------------------- | ------------- | ---------- | +| ✅ [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) | Retrieves the contents of an object. | **Non-compliant behavior:** For non-existent objects, Ozone may return a generic 404 without the structured XML error body defined by AWS S3. | +| ✅ [HeadObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) | Retrieves metadata for an object. | Similar to GET Object, error responses for missing objects might lack detailed AWS-style error XML. | +| ✅ [PutObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) | Uploads a new object. | Supports both single PUT and multipart uploads. **Non-compliant behavior:** Any ACL headers provided in the request are ignored. | +| ✅ [DeleteObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) | Deletes a single object. | Returns standard error codes for missing objects. | +| ✅ [DeleteObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) | Deletes multiple objects in one request. | Bulk delete operation with aggregated error reporting. | +| ✅ [CopyObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) | Copies an object (intra- or inter-bucket). | Supports metadata directives (COPY/REPLACE). **Non-compliant behavior:** Self-copying for metadata updates may differ slightly from AWS S3. | +| ✅ [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) | Lists objects in a bucket with filtering and pagination. | Supports prefix, delimiter, and continuation tokens. | + +### Object Tagging Operations + +| **API Name** | **Feature** | **Note** | +| ------------ | ----------- | -------- | +| ✅ [PutObjectTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html) | Adds or updates tags on an object. | Fully supported; updates replace any existing tag set. | +| ✅ [GetObjectTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html) | Retrieves the tag set of an object. | Returns the tag set as defined by the S3 API. | +| ✅ [DeleteObjectTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html) | Removes the tag set from an object. | Clears all tags associated with the object. | + +### Multipart Upload Operations + +| **API Name** | **Feature** | **Note** | +| ------------ | ----------- | -------- | +| ✅ [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) | Starts a multipart upload session. | Returns an UploadId for subsequent parts. | +| ✅ [UploadPart](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) | Uploads an individual part for a multipart upload. | Part numbering and ETag computation follow AWS semantics. | +| ✅ [ListMultipartUploads](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html) | Lists in-progress multipart uploads in a bucket. | Up to 1000 multipart uploads are returned in a batch by default. | +| ✅ [ListParts](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html) | Lists uploaded parts for an in-progress multipart upload. | Supports pagination if many parts exist. | +| ✅ [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) | Assembles all uploaded parts into a final object. | Returns an ETag that follows AWS multipart rules. | +| ✅ [AbortMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html) | Aborts an in-progress multipart upload. | Frees up storage used by uploaded parts. Incomplete multipart uploads older than 7 days are automatically aborted by Ozone. | + +### Additional Operations + +| **API Name** | **Feature** | **Note** | +| -------------------------- | ------------- | ---------- | +| ✅ Generate Presigned URL | Generates a temporary URL for accessing an object. | Uses AWS Signature V4. **Non-compliant behavior:** The generated URL may include a fixed default region rather than dynamically reflecting the bucket’s location. Ozone now supports generating presigned URLs for all major S3 operations, including `GetObject`, `PutObject`, `DeleteObject`, `HeadObject`, `HeadBucket`, `MultipartUpload`. | + +:::note Additional Notes + +- **Advanced Features Not Supported:** + - **ACLs, Bucket Policies, CORS Configuration, and Website Hosting:** These are not fully implemented; Ozone uses an internal permission model that deviates from AWS S3. + - **Bucket Versioning, Object Locking, Server-Side Encryption, and S3 Select:** These features are currently not supported. + - **Conditional Requests:** Support for conditional requests (e.g., `If-Match`, `If-None-Match`) is planned and tracked in [HDDS-13117](https://issues.apache.org/jira/browse/HDDS-13117). + - **Lifecycle configuration, cross region replication, S3 event notifications** are being implemented and in the roadmap. +- While Ozone S3 Gateway provides extensive support for common S3 operations, users should be aware of the above non-compliant behaviors and limitations when integrating with applications expecting full AWS S3 functionality. +- While Ozone S3 Gateway does not support S3 Server-Side Encryption, it does support encrypted buckets using Apache Ranger KMS. For more information, see the [Transparent Data Encryption](../../05-administrator-guide/02-configuration/03-security/05-encryption/02-transparent-data-encryption.md) documentation. +::: + +## Security + +If security is not enabled, you can *use* **any** AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY +If security is enabled, you can get the key and the secret with the `ozone s3 getsecret` command (*Kerberos based authentication is required). Review Comment: ```suggestion If security is enabled, you can get the key and the secret with the `ozone s3 getsecret` command (*Kerberos based authentication is required). ``` Break out the two paragraphs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
