[
https://issues.apache.org/jira/browse/HDDS-12123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-12123:
-------------------------------
Affects Version/s: 2.0.0
> PutObject using the latest awscli later than or equal to 1.37.0 is fail
> -----------------------------------------------------------------------
>
> Key: HDDS-12123
> URL: https://issues.apache.org/jira/browse/HDDS-12123
> Project: Apache Ozone
> Issue Type: Bug
> Components: s3gateway
> Affects Versions: 1.4.0, 2.0.0, 1.4.1
> Environment: * awscli >= 1.37.0 (aws-cli/1.37.3 Python/3.10.12
> Linux/5.15.146.1-microsoft-standard-WSL2 botocore/1.36.3)
> * Ozone 1.4.1 [compatible
> version|https://github.com/pfnet/ozone/commits/pfn-ozone-1.4.990101/] (based
> on ozone-1.4.1 tag + our internal patches)
> * Secure mode is enabled w/ Kerberos
> * w/o Ranger
> * ozone.http.policy = HTTP_AND_HTTPS
> Reporter: Kohei Sugihara
> Assignee: Ivan Andika
> Priority: Major
>
> S3G returns internal server errors (500) when we upload some files using
> awscli >= 1.37.0 (not AWS CLI version 2), the latest version of AWS CLI
> released in January 2025.
> {code:java}
> 2025-01-22 14:31:36,809 [qtp1327871893-36] WARN
> org.eclipse.jetty.server.HttpChannel: /ksugihara/bin-1m.dat
> javax.servlet.ServletException: javax.servlet.ServletException:
> java.lang.IllegalStateException: Expected: 0 and actual 1048622 write sizes
> do not match
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:516)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at
> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
> at
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
> at
> org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at
> org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> at java.base/java.lang.Thread.run(Thread.java:829) {code}
> This happened on the S3 endpoint enabled with the secure mode. To explore
> this issue in more detail, we changed the ozone. HTTP.policy to
> HTTP_AND_HTTPS to check both endpoints. Curiously, this error does not occur
> on the HTTP endpoint but only on the HTTPS endpoint (port: 9879) in S3G. OM
> and SCM did not output failure logs in their stdout or audit logs, so I think
> this issue is closed within the S3G component.
> {code:java}
> % aws --version
> aws-cli/1.37.3 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2
> botocore/1.36.3
> % aws s3 --endpoint http://<s3g-endpoint>:9878/ cp ./bin-1m.dat
> s3://ksugihara/bin-1m.dat
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp ./bin-1m.dat
> s3://ksugihara/bin-1m.dat
> upload failed: ./bin-1m.dat to s3://ksugihara/bin-1m.dat An error occurred
> (500) when calling the PutObject operation (reached max retries: 4): Internal
> Server Error {code}
> Once downgrading the awscli package earlier than 1.37.0, the bug seems not
> occurs.
> {code:java}
> % pip3 install --upgrade 'awscli<1.37.0'
> % aws --version
> aws-cli/1.36.40 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2
> botocore/1.35.99
> % aws s3 --endpoint http://<s3g-endpoint>:9878/ cp ./bin-1m.dat
> s3://ksugihara/bin-1m.dat
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp ./bin-1m.dat
> s3://ksugihara/bin-1m.dat
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat{code}
> Our current workarounds for using S3 endpoint:
> * Use earlier than awscli 1.37.0: pip3 install 'awscli<1.37.0'
> * Use the HTTP endpoint instead of the HTTPS endpoint
> I don't make sure this information is relavant to identify the root cause,
> awscli v1.37.x is also failure for file uploading to Apache Ozone 1.4{+}.0{+}
> cluster, but in this case S3G returns a conflict error when file is being
> overwritten. New upload that does not overwrite existing files seems to
> success.
> {code:java}
> % aws --version
> aws-cli/1.37.3 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2
> botocore/1.36.3
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp s3://ksugihara/bin-1m.dat
> .
> # overwrite upload
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp ./bin-1m.dat
> s3://ksugihara/bin-1m.dat
> upload failed: ./bin-1m.dat to s3://ksugihara/bin-1m.dat An error occurred
> (Conflict) when calling the PutObject operation: Cannot overwrite file with
> directory
> # new upload
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ --profile pfs0 cp
> ./bin-1m.dat s3://ksugihara/bin-1m.dat.1
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat.1{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
