ivandika3 commented on code in PR #10166:
URL: https://github.com/apache/ozone/pull/10166#discussion_r3194798110
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1880,6 +1880,37 @@ public OzoneKeyDetails getS3KeyDetails(String
bucketName, String keyName,
return getOzoneKeyDetails(keyInfo);
}
+ @Override
+ public OzoneKeyDetails getS3KeyDetails(String bucketName, String keyName,
+ int partNumber, long startOffset,
+ long endOffset) throws IOException {
Review Comment:
Updated thanks.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java:
##########
@@ -647,9 +647,47 @@ private OmKeyInfo readKeyInfo(OmKeyArgs args, BucketLayout
bucketLayout)
.sum());
}
}
+ if (args.hasByteRange()) {
+ filterKeyLocationsByByteRange(value, args.getByteRangeStart(),
+ args.getByteRangeEnd());
+ }
return value;
}
+ private void filterKeyLocationsByByteRange(OmKeyInfo keyInfo,
+ long byteRangeStart, long byteRangeEnd) {
+ OmKeyLocationInfoGroup latestLocationVersion =
+ keyInfo.getLatestVersionLocations();
+ if (latestLocationVersion == null) {
+ return;
+ }
+
+ List<OmKeyLocationInfo> currentLocations =
+ latestLocationVersion.getBlocksLatestVersionOnly();
+ List<OmKeyLocationInfo> filteredLocations = new ArrayList<>();
+ long blockStart = 0;
+ long firstBlockStart = 0;
+ for (OmKeyLocationInfo locationInfo : currentLocations) {
+ long blockEnd = blockStart + locationInfo.getLength() - 1;
+ if (blockStart > byteRangeEnd) {
+ break;
+ }
+ if (blockEnd >= byteRangeStart) {
+ if (filteredLocations.isEmpty()) {
+ firstBlockStart = blockStart;
+ }
+ filteredLocations.add(locationInfo);
+ }
+ blockStart += locationInfo.getLength();
+ }
+
+ keyInfo.updateLocationInfoList(filteredLocations,
+ latestLocationVersion.isMultipartKey(), true);
+ if (!filteredLocations.isEmpty()) {
+ keyInfo.setByteRangeStartOffset(byteRangeStart - firstBlockStart);
+ }
Review Comment:
Good catch, updated.
##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1101,6 +1101,11 @@ message KeyArgs {
// the given ETag for the operation to succeed. This is used for
// S3 conditional writes with the If-Match header.
optional string expectedETag = 24;
+
+ // Byte range requested by S3 GET. OM uses it to return only the
+ // key locations needed to satisfy the read.
+ optional uint64 byteRangeStart = 25;
+ optional uint64 byteRangeEnd = 26;
Review Comment:
Thanks, updated.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java:
##########
@@ -178,6 +179,49 @@ OzoneKeyDetails getS3KeyDetails(String bucketName, String
keyName,
int partNumber)
throws IOException;
+ /**
+ * Get OzoneKey in S3 context with content positioned at the requested
+ * byte range start.
+ * @param bucketName Name of the Bucket
+ * @param keyName Key name
+ * @param partNumber Multipart-upload part number, or zero for the whole key
+ * @param startOffset inclusive byte range start offset
+ * @param endOffset inclusive byte range end offset
+ * @return {@link OzoneKey}
+ * @throws IOException
+ */
+ default OzoneKeyDetails getS3KeyDetails(String bucketName, String keyName,
Review Comment:
This exists as part of a client-side compatibility for other clients that
implements `ClientProtocol`, but it's not `RpcClient`. If `getS3KeyDetails` API
is called on older implementation (that does not have a concrete implementation
of `getS3KeyDetails`) without, the API call will not work since it does not
have any concrete implementation.
However, I think we can safely assume that only `RpcClient` implements
`ClientProtocol` as Ozone is not a library per se (in contrast to Hadoop
`FileSystem` interface which is extended depending on different backing store).
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java:
##########
@@ -658,6 +663,11 @@ private GetKeyInfoResponse getKeyInfo(GetKeyInfoRequest
request,
keyArgs.getForceUpdateContainerCacheFromSCM())
.setMultipartUploadPartNumber(keyArgs.getMultipartNumber())
.build();
+ if (keyArgs.hasByteRangeStart() && keyArgs.hasByteRangeEnd()) {
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
