Kohei Sugihara created HDDS-5271:
------------------------------------
Summary: S3G: Upload with Content-Type: plain/text; charset=utf-8
header will fail
Key: HDDS-5271
URL: https://issues.apache.org/jira/browse/HDDS-5271
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode, S3
Affects Versions: 1.1.0
Reporter: Kohei Sugihara
Attachments: s3g.log, upload.go
Upload content via S3 gateway with "Content-Type: plain/text; charset=utf-8"
header will fail with 500 code. At the same time, "Content-Type: plain/text"
(without charset) and "Content-Type: plain/text; charset=UTF-8" (in uppercase)
were accepted.
We're running the official Ozone v1.1.0 release.
These lines shown as below are known behavior:
* 'text/plain' ==> OK
* 'text/plain; charset=UTF-8' ==> OK
* 'text/plain; charset=utf-8' ==> 500
* 'text/html; charset=UTF-8' ==> OK (same as text/plain case)
* 'text/html; charset=utf-8' ==> 500 (same as text/plain case)
* 'text/markdown; charset=UTF-8' ==> OK
* 'text/markdown; charset=utf-8' ==> OK
I've attached a code snippet (upload.go) and a log from S3 gateway (s3g.log).
This code can run easily with following command lines.
{code:none}
% go run upload.go --contenttype='text/plain; charset=utf-8'
text/plain; charset=utf-8
InternalServerError: Internal Server Error
status code: 500, request id: , host id: exit status 1
% go run upload.go --contenttype='text/plain; charset=UTF-8'
text/plain; charset=UTF-8
&\{http://example.com/<BUCKET>/<KEY> <nil> <nil>}%
{code}
HTTP request from upload.go is here:
{code:none}
PUT /<BUCKET>/<KEY> HTTP/1.1
Host: localhost:9999
User-Agent: aws-sdk-go/1.38.46 (go1.16.4; linux; amd64) S3Manager
Content-Length: 4
Authorization: AWS4-HMAC-SHA256
[email protected]/20210525/dummy/s3/aws4_request,
SignedHeaders=content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date,
Signature=<HASH>
Content-Md5: AzvZSxFo1+Tw1kTDyV41vw==
Content-Type: text/plain; charset=utf-8
X-Amz-Content-Sha256: <HASH>
X-Amz-Date: 20210525T080147Z
Accept-Encoding: gzip
TEST^C
{code}
h2. Investigation
We thought the issue is similar to HDDS-4856 but this issue will happen when
empty Content-Type header was given. Our facing problem will appear when
Content-Type defined together with the charset directive.
The stack trace of S3 gateway indicates the request proceeded to remote
invocation and failed with authentication error with
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken).
There are many parts to preprocess a request before reaching RPC Client, so I
suspect some Java servlet filters like ozone.s3.HeaderPreprocessor and
hdds.server.http.NoCacheFilter. But as far as I found out, these are not a
cause of this problem directly (because these are almost for generating
response header). I also suspected some components like OzoneClientProducer,
Signature Processor and HeaderPreprocessor. I think this is not direct cause.
How do I investigate this problem in more detail?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
