jmahonin commented on PR #4753: URL: https://github.com/apache/hadoop/pull/4753#issuecomment-1224304355
It looks like the issue is in retrieving `s3a://landsat-pds/scene_list.gz`. The corresponding role should have full S3 access at this point. I've enabled the low-level request tracing. Here's what it looks like for the first parameterized test: `testJobSubmissionCollectsTokens[0](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob)` ``` 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "HEAD /scene_list.gz HTTP/1.1[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Host: landsat-pds.s3.amazonaws.com[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-invocation-id: <redacted>[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-request: attempt=1;max=21[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=<redacted>/20220823/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;referer;user-agent;x-amz-content-sha256;x-amz-date, Signature=<redacted>[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Content-Type: application/octet-stream[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Referer: https://audit.example.org/hadoop/1/op_get_file_status/<redacted>/?op=op_get_file_status&p1=scene_list.gz&pr=jmahonin/[email protected]&ps=<redacted>&id=<redacted>&t0=15&fs=<redacted>&t1=15&ts=1661271325482[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.262 Linux/5.10.104-linuxkit OpenJDK_64-Bit_Server_VM/25.342-b07 java/1.8.0_342 kotlin/1.4.10 vendor/Private_Build cfg/retry-mode/legacy[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "X-Amz-Date: 20220823T161525Z[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "HTTP/1.1 400 Bad Request[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "x-amz-request-id: ZMY1169S2RC7NEHP[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "x-amz-id-2: ENtHr5DR7HX+qkh5FfYQublaU82ykB/SD5fAvR5kC6JZJibFVLH6Rq+F/EutrE3dAL1uTz6yad8=[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Content-Type: application/xml[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Date: Tue, 23 Aug 2022 16:15:25 GMT[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Server: AmazonS3[\r][\n]" 2022-08-23 16:15:25,937 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Connection: close[\r][\n]" 2022-08-23 16:15:25,937 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "[\r][\n]" ``` I've also tried adding this to my auth-keys, which likewise does not work: ``` <property> <name>fs.s3a.bucket.landsat-pds.aws.credentials.provider</name> <value>org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider</value> </property> ``` I've seen mention in HADOOP-13551 and HADOOP-18340 that folks have run into similar issues with this test, although I haven't been able to determine what, if anything, was done to correct it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
