[
https://issues.apache.org/jira/browse/HADOOP-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17575854#comment-17575854
]
Ahmar Suhail commented on HADOOP-18353:
---------------------------------------
This is happening because the HEAD responses don't have a body. Since the S3
API sends error codes + messages in the body, when you make a GET request, S3
returns ExpiredToken in the body, with statusCode 400 and statusText: Bad
Request. For HEAD, there is no body, and statusCode is 400, and statusText is
Bad Request (so same as GET). For HEAD, the SDK then tries to build the
exception from the headers, see
[S3ErrorResponseHandler|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/internal/S3ErrorResponseHandler.java#L87].
When building the exception, it just copies whatever is in statusText, which
is why you get Bad Request in the exception.
[~mthakur] is it ok to resolve this or is there anything else we want to do
here?
> HEAD OBJECT returns only 400 BAD REQUEST when token is expired
> --------------------------------------------------------------
>
> Key: HADOOP-18353
> URL: https://issues.apache.org/jira/browse/HADOOP-18353
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.3
> Reporter: Mukund Thakur
> Priority: Major
>
> I tried reproducing this today by changing this test
> [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ATemporaryCredentials.java#L116]
> . Getting a session token for 15 mins and trying every one mins and finally
> it fails after 15 mins.
>
> Looks like the AWS SDK is not having the Expired Token error message as I
> could see the same in access logs but I see BadRequest on the SDK logs.
>
> *S3A Connector logs with SDK debug enabled.*
> 2022-07-13 15:44:15,318 [JUnit-testSTS] DEBUG s3a.AWSCredentialProviderList
> (AWSCredentialProviderList.java:getCredentials(184)) - Using credentials from
> TemporaryAWSCredentialsProvider
> 2022-07-13 15:44:15,319 [JUnit-testSTS] DEBUG amazonaws.request
> (AmazonHttpClient.java:executeOneRequest(1285)) - Sending Request: HEAD
> [https://mthakur-us-west-1.s3.us-west-1.amazonaws.com|https://mthakur-us-west-1.s3.us-west-1.amazonaws.com/]
> /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e Headers:
> (amz-sdk-invocation-id: 41e6e504-1c2b-2701-09bb-ae692dff2515, Content-Type:
> application/octet-stream, Referer:
> [https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318],
> User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7
> Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10
> vendor/Oracle_Corporation cfg/retry-mode/legacy, )
> 2022-07-13 15:44:15,623 [JUnit-testSTS] DEBUG amazonaws.request
> (AmazonHttpClient.java:handleErrorResponse(1846)) - \{*}Received error
> response: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request
> (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request
> ID{*}: WMGQ0KC4MHEMZTQC; S3 Extended Request ID:
> IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=;
> Proxy: null), S3 Extended Request ID:
> IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
> 2022-07-13 15:44:15,624 [JUnit-testSTS] DEBUG s3a.S3AFileSystem
> (S3AFileSystem.java:close(3814)) - Filesystem s3a://mthakur-us-west-1 is
> closed
> *AWS access logs*
> 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
> mthakur-us-west-1 [13/Jul/2022:20:44:15 +0000] 67.79.115.98 -
> WMGQ0KC4MHEMZTQC REST.HEAD.OBJECT
> test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e "HEAD
> /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e HTTP/1.1" *400
> ExpiredToken* 556 - 5 -
> "[https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318]";
> "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7
> Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10
> vendor/Oracle_Corporation cfg/retry-mode/legacy" -
> IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
> SigV4 ECDHE-RSA-AES128-SHA AuthHeader
> [mthakur-us-west-1.s3.us-west-1.amazonaws.com|http://mthakur-us-west-1.s3.us-west-1.amazonaws.com/]
> TLSv1.2 -
>
> I tested by running repeatedly ITestCustomSigner in S3A, and also just
> ListObjectsV2 on loop… I did just notice your test is failing with HEAD, and
> *I can reproduce* by running this after credential expiry.
>
> aws s3api head-object --bucket djonesoa-us-west-2 --region us-west-2 --key
> test-object –debug
>
> To summarise:
> * If I run ListObjectsV2, I get “400 ExpiredToken”{+}{+}{+}{+}
> * If I run HeadObject, I get “400 Bad Request”{+}{+}{+}{+}
> * If I run GetObject, I get “400 ExpiredToken”
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
