Huang Kuan Hao created HDDS-17486:
-------------------------------------
Summary: Migrate existing S3 object read APIs to S3ClientProtocol
delegation
Key: HDDS-17486
URL: https://issues.apache.org/jira/browse/HDDS-17486
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Client, s3gateway
Reporter: Huang Kuan Hao
HDDS-17452 introduced {{S3ClientProtocol}} as a delegating {{ClientProtocol}}
implementation for S3 Gateway. It preserves existing behavior, including
forwarding calls to the S3-specific read methods in {{ClientProtocol}} and
{{RpcClient}}.
The [discussion in PR
#11740|https://github.com/apache/ozone/pull/11740#pullrequestreview-5638868061]
proposed using this delegation layer to avoid maintaining parallel generic and
S3-specific operation APIs. A [follow-up discussion in PR
#11757|https://github.com/apache/ozone/pull/11757#issuecomment-6419318745]
raised migrating the existing S3 read APIs and considering deprecation of the
old entry points.
This issue tracks that migration for GetObject, HeadObject, and
GetObjectAttributes, including requests for a specific multipart part. The goal
is to consolidate the client implementation while preserving existing behavior
and RPC savings. These read paths already support resolving the S3 volume
within the metadata request; migration must not reintroduce a separate volume
lookup where none is currently needed.
h3. Scope
Review and migrate these {{ClientProtocol}} methods and their S3 Gateway
callers:
||Existing method||Current usage||
|{{getS3KeyDetails(bucketName, keyName)}}|GetObject|
|{{getS3KeyDetails(bucketName, keyName, partNumber)}}|GetObject for a specified
part|
|{{headS3Object(bucketName, keyName)}}|HeadObject and GetObjectAttributes
without ObjectParts|
|{{headS3Object(bucketName, keyName, partNumber)}}|HeadObject for a specified
part|
|{{headS3ObjectAttributes(bucketName, keyName)}}|GetObjectAttributes requesting
ObjectParts|
The main implementation areas are {{ClientProtocol}}, {{S3ClientProtocol}},
{{RpcClient}}, {{ObjectEndpoint}}, {{ObjectAttributesHandler}}, and their
relevant tests and stubs. {{RpcClient#getS3KeyInfo}} is a private helper, not a
public {{ClientProtocol}} method to deprecate.
h3. Proposed approach
# Define the replacement calls. Reuse generic {{ClientProtocol}} operations
where they provide the required behavior. Basic reads already have
{{getKeyDetails(volumeName, bucketName, keyName)}} and {{headObject(volumeName,
bucketName, keyName)}}. Part-specific reads and ObjectAttributes do not
currently have equivalent generic signatures; agree on the minimal API changes
needed to retain their functionality.
# Use {{S3ClientProtocol}} for S3-specific delegation and applicable
compatibility handling, while sharing request construction and response
handling in {{RpcClient}}. Avoid another parallel set of S3 operation methods
or duplicate RPC implementations.
# Migrate the S3 Gateway callers, including both GetObjectAttributes paths, and
update affected stubs and mocks.
# Decide which existing public methods require compatibility adapters, whether
to mark them {{@Deprecated}}, and document their replacements. Retained entry
points should share the replacement implementation.
These steps may be delivered through separate PRs under this issue.
h3. Behavior to preserve
* *Volume resolution and RPC count:* retain the metadata-read path without an
additional {{GetS3VolumeContext}} call wherever it is already supported.
Existing read capabilities must not become dependent solely on the newer
capability used by other S3 operations.
* *Principal propagation:* preserve updating the request's S3 principal from
the OM response, including its use by KMS when reading encrypted objects.
* *GET:* preserve content, metadata, range handling, and opening the input
stream using the returned key details.
* *HEAD:* preserve metadata-only processing, including avoiding unnecessary
block-token generation and pipeline refresh.
* *Part-specific reads:* preserve selection, validation, returned size, error
mapping, and the existing fallback behavior for older OM versions.
* *GetObjectAttributes:* preserve completed multipart part sizes, bucket layout
information, and response pagination. Do not add a bucket lookup or bucket READ
requirement to obtain information already returned by the metadata operation.
* *Authorization and errors:* preserve S3 action context, expected-owner
handling, and existing error translation.
* *Generic clients:* retain explicit-volume read behavior for non-S3 callers.
* *Metadata reuse:* preserve HDDS-16658's CopyObject source-key reuse if shared
read implementation changes affect that path. CopyObject already uses generic
{{getKeyDetails}} and does not itself require a legacy S3 method rename.
h3. Compatibility decisions
Some methods in this scope are already present in released clients. [Ozone
2.2.1's
ClientProtocol|https://github.com/apache/ozone/blob/ozone-2.2.1/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java#L157]
includes basic {{headS3Object}} and both {{getS3KeyDetails}} overloads.
Before removing methods, agree on:
* Which entry points require compatibility adapters.
* Whether and when to add {{@Deprecated}}.
* Replacement APIs for part-specific reads and ObjectAttributes.
* Any later release in which deprecated methods may be removed.
Migrating S3 Gateway callers and removing public APIs can be separate steps.
This issue does not assume all existing read APIs can be removed before 2.3.0;
the deprecation policy and removal schedule remain to be agreed.
h3. Validation and completion criteria
* The listed S3 Gateway callers use the agreed replacement operations.
* Retained compatibility methods share the implementation rather than duplicate
RPC logic.
* Relevant tests cover ordinary GET/HEAD, part-specific reads and invalid
parts, and GetObjectAttributes with and without ObjectParts.
* Tests verify no added volume or bucket metadata RPCs on paths that currently
avoid them.
* Relevant coverage verifies principal propagation, authorization, supported
older OM behavior, and explicit-volume reads.
* Existing metadata reuse remains intact where shared code changes affect it.
* Deprecation and removal decisions are documented; deferred removal has an
explicit follow-up.
h3. Related issues
* HDDS-16602: related S3 RPC optimization work.
* HDDS-17452: shared client delegation.
* HDDS-7324: S3 reads without a preliminary volume lookup.
* HDDS-9654: GET for a specified part.
* HDDS-15740: part validation for GET/HEAD.
* HDDS-16073: ObjectParts listing and pagination.
* HDDS-16658: CopyObject source metadata reuse.
AbortMultipartUpload migration is tracked separately. Pending operation PRs
continue to handle their own delegation changes in their existing issues.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
