[
https://issues.apache.org/jira/browse/HDDS-17644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18136333#comment-18136333
]
Andy Huang commented on HDDS-17644:
-----------------------------------
Reproduced the JIRA steps against both versions using separate, fresh local
Docker Compose clusters (SCM, OM, one datanode, and S3 Gateway).
* Before: {{ec5ed5f7ea0e4b406aee53aae4906616819cba2b}} (parent of the fix).
* After: {{0fc9b81bf389aa3b50c728a1d05ebf31300844a3}} (PR #11900).
* Both distributions built successfully with JDK 21 using {{mvn -B -pl
:ozone-dist -am package -DskipTests -DskipShade -DskipRecon}}. Runtime image:
{{apache/ozone-runner:20260922-1-jdk25}}. Default local development
configuration, replication RATIS/ONE.
* Requests below use dummy credentials {{hadoop:hadoop}} for this local
development cluster.
In each cluster, created the bucket explicitly with FSO layout:
{code:bash}
ozone sh bucket create --layout FILE_SYSTEM_OPTIMIZED --type RATIS
--replication ONE /s3v/hdds17644
ozone sh bucket info /s3v/hdds17644
{code}
The bucket info output confirmed {{"bucketLayout" : "FILE_SYSTEM_OPTIMIZED"}}
in both runs.
The following are captured command/output excerpts. HTTP status and response
bodies are preserved; routine HTTP headers are omitted.
h3. Before the fix
{code}
# create-parent
$ curl -sS -i --max-time 60 --aws-sigv4 aws:amz:us-east-1:s3 --user
hadoop:hadoop -X PUT --data-binary 'parent
' http://localhost:9878/hdds17644/file-062844
HTTP/1.1 200 OK
# put-child
$ curl -sS -i --max-time 60 --aws-sigv4 aws:amz:us-east-1:s3 --user
hadoop:hadoop -X PUT --data-binary 'child
' http://localhost:9878/hdds17644/file-062844/child
HTTP/1.1 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InvalidRequest</Code>
<Message>An error occurred (InvalidRequest) when calling the PutObject/MPU
PartUpload operation: ozone.om.enable.filesystem.paths is enabled Keys are
considered as Unix Paths. Path has Violated FS Semantics which caused put
operation to fail.</Message>
<Resource>file-062844/child</Resource>
<RequestId>58d96690-251e-443a-80f3-01e0dc790353</RequestId>
<HostId>eVU0Kx0Snk</HostId>
</Error>
# initiate
$ curl -sS -i --max-time 60 --aws-sigv4 aws:amz:us-east-1:s3 --user
hadoop:hadoop -X POST
'http://localhost:9878/hdds17644/file-062844/child?uploads'
HTTP/1.1 200 OK
<?xml version="1.0" encoding="UTF-8"
standalone="yes"?><InitiateMultipartUploadResult
xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Bucket>hdds17644</Bucket><Key>file-062844/child</Key><UploadId>01a19b4a-a43e-70e9-ab40-2fcd0e6d3013-117545810173755394</UploadId></InitiateMultipartUploadResult>
Returned UploadId: 01a19b4a-a43e-70e9-ab40-2fcd0e6d3013-117545810173755394
# upload-part
$ curl -sS -i --max-time 60 --aws-sigv4 aws:amz:us-east-1:s3 --user
hadoop:hadoop -X PUT --data-binary 'part
'
'http://localhost:9878/hdds17644/file-062844/child?uploadId=01a19b4a-a43e-70e9-ab40-2fcd0e6d3013-117545810173755394&partNumber=1'
HTTP/1.1 500 Server Error
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InternalError</Code>
<Message>We encountered an internal error. Please try again.</Message>
<Resource>file-062844/child</Resource>
<RequestId>731e4c4f-2081-4148-b6f3-55a6bb1752cb</RequestId>
<HostId>hyMcclukrnh3XiZ</HostId>
</Error>
# list-parts
$ curl -sS -i --max-time 60 --aws-sigv4 aws:amz:us-east-1:s3 --user
hadoop:hadoop -X GET
'http://localhost:9878/hdds17644/file-062844/child?uploadId=01a19b4a-a43e-70e9-ab40-2fcd0e6d3013-117545810173755394'
HTTP/1.1 200 OK
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><ListPartsResult
xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Bucket>hdds17644</Bucket><Key>file-062844/child</Key><UploadId>01a19b4a-a43e-70e9-ab40-2fcd0e6d3013-117545810173755394</UploadId><StorageClass>REDUCED_REDUNDANCY</StorageClass><PartNumberMarker>0</PartNumberMarker><NextPartNumberMarker>0</NextPartNumberMarker><MaxParts>1000</MaxParts><IsTruncated>false</IsTruncated></ListPartsResult>
# list-uploads
$ curl -sS -i --max-time 60 --aws-sigv4 aws:amz:us-east-1:s3 --user
hadoop:hadoop -X GET 'http://localhost:9878/hdds17644?uploads'
HTTP/1.1 200 OK
<?xml version="1.0" encoding="UTF-8"
standalone="yes"?><ListMultipartUploadsResult
xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Bucket>hdds17644</Bucket><NextKeyMarker></NextKeyMarker><NextUploadIdMarker></NextUploadIdMarker><MaxUploads>1000</MaxUploads><IsTruncated>false</IsTruncated><Upload><Key>file-062844/child</Key><UploadId>01a19b4a-a43e-70e9-ab40-2fcd0e6d3013-117545810173755394</UploadId><Owner><DisplayName>ozone</DisplayName><ID>bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93</ID></Owner><Initiator><DisplayName>ozone</DisplayName><ID>bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93</ID></Initiator><StorageClass>REDUCED_REDUNDANCY</StorageClass><Initiated>2026-11-02T08:05:22.622Z</Initiated></Upload></ListMultipartUploadsResult>
# abort-upload
$ curl -sS -i --max-time 60 --aws-sigv4 aws:amz:us-east-1:s3 --user
hadoop:hadoop -X DELETE
'http://localhost:9878/hdds17644/file-062844/child?uploadId=01a19b4a-a43e-70e9-ab40-2fcd0e6d3013-117545810173755394'
HTTP/1.1 204 No Content
{code}
S3 Gateway also logged the underlying OM error during the failed UploadPart:
{code}
NOT_A_FILE org.apache.hadoop.ozone.om.exceptions.OMException: Can not create
file: file-062844/child as there is already file in the given path
{code}
Initiation returned an UploadId even though the parent was a file. UploadPart
then failed with HTTP 500 InternalError. ListParts contained no parts, while
ListMultipartUploads still contained the unusable upload. The final DELETE
above aborted it after capturing the results.
h3. After the fix
{code}
# create-parent
$ curl -sS -i --max-time 60 --aws-sigv4 aws:amz:us-east-1:s3 --user
hadoop:hadoop -X PUT --data-binary 'parent
' http://localhost:9878/hdds17644/file-062844
HTTP/1.1 200 OK
# put-child
$ curl -sS -i --max-time 60 --aws-sigv4 aws:amz:us-east-1:s3 --user
hadoop:hadoop -X PUT --data-binary 'child
' http://localhost:9878/hdds17644/file-062844/child
HTTP/1.1 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InvalidRequest</Code>
<Message>An error occurred (InvalidRequest) when calling the PutObject/MPU
PartUpload operation: ozone.om.enable.filesystem.paths is enabled Keys are
considered as Unix Paths. Path has Violated FS Semantics which caused put
operation to fail.</Message>
<Resource>file-062844/child</Resource>
<RequestId>3efaba06-ff54-46a9-99d6-5ee44826a551</RequestId>
<HostId>mX3HsrvvqNGY0SB</HostId>
</Error>
# initiate
$ curl -sS -i --max-time 60 --aws-sigv4 aws:amz:us-east-1:s3 --user
hadoop:hadoop -X POST
'http://localhost:9878/hdds17644/file-062844/child?uploads'
HTTP/1.1 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InvalidRequest</Code>
<Message>Invalid Request</Message>
<Resource>file-062844/child</Resource>
<RequestId>b531f940-95bb-4548-a7d8-783c9067221a</RequestId>
<HostId>W0vTLUjlh</HostId>
</Error>
No UploadId was returned; UploadPart is not attempted.
# list-uploads
$ curl -sS -i --max-time 60 --aws-sigv4 aws:amz:us-east-1:s3 --user
hadoop:hadoop -X GET 'http://localhost:9878/hdds17644?uploads'
HTTP/1.1 200 OK
<?xml version="1.0" encoding="UTF-8"
standalone="yes"?><ListMultipartUploadsResult
xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Bucket>hdds17644</Bucket><NextKeyMarker></NextKeyMarker><NextUploadIdMarker></NextUploadIdMarker><MaxUploads>1000</MaxUploads><IsTruncated>false</IsTruncated></ListMultipartUploadsResult>
{code}
PutObject continues to return HTTP 400 InvalidRequest. CreateMultipartUpload
now also returns HTTP 400 InvalidRequest immediately. No UploadId is returned,
so the UploadPart step cannot be performed for this rejected initiation.
ListMultipartUploads confirms that no incomplete upload was created.
> S3 CreateMultipartUpload is accepted under a parent that is a file, then
> every UploadPart fails with 500 InternalError
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17644
> URL: https://issues.apache.org/jira/browse/HDDS-17644
> Project: Apache Ozone
> Issue Type: Bug
> Components: S3
> Reporter: Bablu Raul
> Assignee: Andy Huang
> Priority: Major
> Labels: pull-request-available
>
> In an FSO bucket, if a parent path already exists as a regular file, a
> {{PutObject}} request to create an object under that file is correctly
> rejected with HTTP 400 {{{}InvalidRequest{}}}.
> However, {{CreateMultipartUpload}} for the same key succeeds and returns an
> {{{}UploadId{}}}. When attempting to upload a part, the request fails with
> HTTP 500 {{{}InternalError{}}}.
> The S3 Gateway logs show that Ozone Manager rejects the operation with
> {{NOT_A_FILE}} because the parent path is already an existing file. However,
> the S3 Gateway returns a generic 500 error instead of an appropriate 4xx
> error.
> As a result, the multipart upload is created but no parts can be uploaded,
> leaving it unusable until explicitly aborted.
> *Steps to Reproduce*
> # Create a bucket with {{FILE_SYSTEM_OPTIMIZED}} layout.
> # Create a regular file in the bucket (for example, {{{}file-062844{}}}).
> # Try a {{PutObject}} request using {{file-062844/child}} as the key. It
> should fail with HTTP 400 {{{}InvalidRequest{}}}.
> # Initiate a multipart upload for the same key using
> {{{}CreateMultipartUpload{}}}.
> # Verify that the request succeeds and returns an {{{}UploadId{}}}.
> # Attempt to upload a part using the returned {{{}UploadId{}}}.
> *Actual Result*
> * {{CreateMultipartUpload}} succeeds even though the parent path is an
> existing file.
> * Every {{UploadPart}} request fails with HTTP 500 {{{}InternalError{}}}.
> * Ozone Manager reports a {{NOT_A_FILE}} exception, but the S3 Gateway
> returns a generic 500 error.
> * No parts are stored, and the multipart upload can only be aborted.
> *Expected Result*
> * {{CreateMultipartUpload}} should be rejected immediately with HTTP 400
> {{{}InvalidRequest{}}}, consistent with {{{}PutObject{}}}.
> * If {{UploadPart}} encounters a {{NOT_A_FILE}} exception, the S3 Gateway
> should return an appropriate 4xx error instead of HTTP 500
> {{{}InternalError{}}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
