chungen0126 commented on code in PR #10006: URL: https://github.com/apache/ozone/pull/10006#discussion_r3193864309
########## hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md: ########## @@ -0,0 +1,142 @@ +--- +title: S3 Multi Chunks Verification +summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G. +date: 2026-04-29 +jira: HDDS-12542 +status: proposed +author: Chung-En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Context & Motivation + +Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead. + +# Goal + +Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms: +- STREAMING-AWS4-HMAC-SHA256-PAYLOAD +- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER +- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD +- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER + +# Proposed Solution + +Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs. + +## Secret Key + +Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data. Review Comment: Thanks for the review. However, under the AWS Signature V4 protocol, the client never transmits the raw Secret Key over the network. They only send the AWSAccessKeyId and the calculated Signature. Since S3G never actually receives the user's Secret Key in the request, it is impossible for S3G to calculate the derived key locally. It must query Ozone Manager (OM), which securely holds the master keys, to perform the derivation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
