ChenSammi commented on code in PR #10006: URL: https://github.com/apache/ozone/pull/10006#discussion_r3232248976
########## hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md: ########## @@ -0,0 +1,142 @@ +--- +title: S3 Multi Chunks Verification +summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G. +date: 2026-04-29 +jira: HDDS-12542 +status: proposed +author: Chung-En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Context & Motivation + +Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead. + +# Goal + +Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms: +- STREAMING-AWS4-HMAC-SHA256-PAYLOAD +- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER +- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD +- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER + +# Proposed Solution + +Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs. + +## Secret Key + +Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data. + +From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits: +- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM. +- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services. + +## HMAC-SHA256 Implementation + +We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i mplementation. + +## ECDSA-SHA256 Implementation + +Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates: Review Comment: I'm not sure if we need to implement ECDSA in Sigv4a. ECDSA is slow compared with sha256. AWS introduced Sigv4a for some special endpoints. We don't have this case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
