shuan1026 opened a new pull request, #11979: URL: https://github.com/apache/ozone/pull/11979
## What changes were proposed in this pull request? A multipart upload can be completed with non-contiguous part numbers, for example 1 and 3. AWS S3 then addresses the parts of the completed object as 1 to N. Ozone keeps the part numbers used during upload: `S3MultipartUploadCompleteRequest` stamps them on the blocks of the completed key, and part-aware reads match on that stored number. For an object completed with parts 1 and 3: | `GetObject` with | Before | After | |---|---|---| | `partNumber=2` | `400 InvalidPart` | data uploaded as part 3, as on AWS S3 | | `partNumber=3` | data uploaded as part 3 | `400 InvalidPart`, the object has 2 parts | `x-amz-mp-parts-count` is 2 in both cases. The AWS SDK for Java v2 reads `partNumber` 1 to N, so a `multipartEnabled` `S3AsyncClient` or `S3TransferManager` cannot download such objects. This patch stamps each block with the 1-based position of its part in the completion list instead. The list is already validated to be strictly ascending, so the order of the data does not change. `GetObject` and `HeadObject` with `partNumber`, and `ObjectParts` of `GetObjectAttributes`, read the stored number and need no change. Keys completed before this change keep their original part numbers. There is no proto or DB schema change. The `206` / `Content-Range` response that the SDK also expects is covered by HDDS-17662. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17663 ## How was this patch tested? - New `TestS3MultipartUploadCompleteRequest#testPartsRenumberedSequentiallyOnComplete`, also run for FSO by `TestS3MultipartUploadCompleteRequestWithFSO`. It completes an upload with parts 1, 3 and 5 and checks that the blocks carry part numbers 1, 2 and 3 in the original data order. It fails with `expected: <[1, 2, 3]> but was: <[1, 3, 5]>` before the production change and passes after it. - `TestOzoneClientMultipartUploadWithFSO#testGetPartNumberWithNonContiguousParts` asserted the old behavior and is updated. It fails with `Cannot read part 2 of key ... because it does not exist` before the production change and passes after it. - `s3/MultipartUpload.robot` read part number 3 of an object completed with parts 1 and 3. It now reads part number 2. Not run locally. - ci: https://github.com/shuan1026/ozone/actions/runs/42367854407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
