[
https://issues.apache.org/jira/browse/HADOOP-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878695#comment-17878695
]
ASF GitHub Bot commented on HADOOP-19258:
-----------------------------------------
diljotgrewal commented on PR #7015:
URL: https://github.com/apache/hadoop/pull/7015#issuecomment-2325536384
Testing against a Regular S3 bucket in `us-east-1`:
auth-keys
```
<configuration>
<property>
<name>test.fs.s3a.name</name>
<value>s3a://<REDACTED>/</value>
</property>
<property>
<name>fs.contract.test.fs.s3a</name>
<value>s3a://<REDACTED>/</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<description>AWS access key ID. Omit for IAM role-based
authentication.</description>
<value><REDACTED></value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<description>AWS secret key. Omit for IAM role-based
authentication.</description>
<value><REDACTED></value>
</property>
<property>
<name>fs.s3a.encryption.key</name>
<value>arn:aws:kms:us-east-1:<REDACTED>:key/<REDACTED></value>
</property>
<property>
<name>fs.s3a.assumed.role.arn</name>
<value>arn:aws:iam::<REDACTED>:role/<REDACTED></value>
</property>
<property>
<name>test.fs.s3a.sts.enabled</name>
<value>false</value>
</property>
</configuration>
```
#### Scale Tests
```
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
```
```
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 3, Time elapsed:
8.89 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[ERROR] testRequestTimeout(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)
Time elapsed: 0.029 s <<< ERROR!
java.lang.IllegalArgumentException: Cannot locate field clientConfiguration
on class
software.amazon.awssdk.services.s3.internal.crossregion.S3CrossRegionSyncClient
at org.apache.commons.lang3.Validate.isTrue(Validate.java:158)
at
org.apache.commons.lang3.reflect.FieldUtils.readField(FieldUtils.java:469)
at
org.apache.hadoop.fs.s3a.ITestS3AConfiguration.getField(ITestS3AConfiguration.java:535)
at
org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testRequestTimeout(ITestS3AConfiguration.java:449)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
```
```
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
169.585 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[ERROR]
testThreadPoolCoolDown(org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps)
Time elapsed: 143.941 s <<< FAILURE!
java.lang.AssertionError: Failed to find threads in active FS - test is
flawed. Actual: 0
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failEquals(Assert.java:187)
at org.junit.Assert.assertNotEquals(Assert.java:201)
at
org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps.testThreadPoolCoolDown(ITestS3AConcurrentOps.java:206)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
```
```
[ERROR] Failures:
[ERROR]
ITestS3AConcurrentOps.testThreadPoolCoolDown:206->Assert.assertNotEquals:201->Assert.failEquals:187->Assert.fail:89
Failed to find threads in active FS - test is flawed. Actual: 0
[ERROR] Errors:
[ERROR] ITestS3AConfiguration.testRequestTimeout:449->getField:535 »
IllegalArgument C...
[INFO]
[ERROR] Tests run: 1327, Failures: 1, Errors: 1, Skipped: 127
```
#### adding access point:
```
<property>
<name>fs.s3a.bucket.sample-bucket.accesspoint.arn</name>
<value> <REDACTED> </value>
<description>Configure S3a traffic to use this AccessPoint</description>
</property>
```
```
mvn clean verify -Dit.test=ITest* -Dtest=none
```
Getting a lot of errors, currently debugging:
several parsing errors
```
grep "Failed to parse b-00" itest_log| sort | uniq -c
132 java.lang.Exception: Failed to parse b-00
```
```
[^[[1;31mERROR^[[m]
testStatistics(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics)
Time elapsed: 2.179 s <<< FAILURE!
java.lang.AssertionError: Mismatch in bytes written expected:<697> but
was:<512>
```
```
[^[[1;31mERROR^[[m]
testClosedInstrumentation(org.apache.hadoop.fs.s3a.ITestS3AClosedFS) Time
elapsed: 0.333 s <<< FAILURE!
org.junit.ComparisonFailure: [S3AInstrumentation.hasMetricSystem()]
expected:<[fals]e> but was:<[tru]e>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
```
few encryption errors:
```
[^[[1;31mERROR^[[m]
testEncryptionOverRename(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)
Time elapsed: 2.502 s <<< FAILURE!
java.lang.AssertionError: Key wrap algo isn't same as expected
expected:<kms+context> but was:<null>
```
```
[^[[1;31mERROR^[[m]
testDirectoryListingFileLengths(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)
Time elapsed: 3.498 s <<< FAILURE!
java.lang.AssertionError:
[File lengths aren't the same as expected from FileStatus dir. listing]
Expecting:
<[0, 1, 239, 4079]>
to contain exactly in any order:
<[0, 1, 255, 4095]>
elements not found:
<[255, 4095]>
and elements not expected:
<[239, 4079]>
```
```
[^[[1;31mERROR^[[m]
testBytesReadWithStream(org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic)
Time elapsed: 1.646 s <<< FAILURE!
java.lang.AssertionError: Mismatch in number of FS bytes read by
InputStreams expected:<2048> but was:<196167>
```
```
[^[[1;31mERROR^[[m]
testCustomSignerAndInitializer[simple-delete](org.apache.hadoop.fs.s3a.auth.ITestCustomSigner)
Time elapsed: 1.138 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on
s3a://s3atestingdsgrewal/job-00/test/testCustomSignerAndInitializer[simple-delete]/customsignerpath1:
software.amazon.awssdk.services.s3.model.S3Exception: The authorization
mechanism you have provided is not supported. Please use Signature Version 4.
(Service: S3, Status Code: 400, Request ID: Z68EWFD946XZRPW9, Extended Request
ID:
4oF/6V21FWJZdR5osr6DcNYC6RLxEQZrNxEf3/X8ur0Tm/aeVe31aNtZXT/VU/yrG17ZMUOAZK4=):InvalidRequest:
The authorization mechanism you have provided is not supported. Please use
Signature Version 4. (Service: S3, Status Code: 400, Request ID:
Z68EWFD946XZRPW9, Extended Request ID:
4oF/6V21FWJZdR5osr6DcNYC6RLxEQZrNxEf3/X8ur0Tm/aeVe31aNtZXT/VU/yrG17ZMUOAZK4=)
```
```
[^[[1;31mERROR^[[m]
testComplexDirActions(org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List)
Time elapsed: 16.345 s <<< FAILURE!
java.lang.AssertionError: getLen expected:<512> but was:<496>
```
```
[^[[1;31mERROR^[[m]
testRenamePopulatesFileAncestors2(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)
Time elapsed: 2.337 s <<< FAILURE!
java.lang.AssertionError: Number of bytes copied: files_copied_bytes
expected:<16384> but was:<16368>
```
```
[^[[1;31mERROR^[[m] ^[[1;31m ITestS3AClosedFS.testClosedInstrumentation:111
[S3AInstrumentation.hasMetricSystem()] expected:<[fals]e> but was:<[tru]e>^[[m
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column:
4235] (through reference chain:
org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])
```
#### ILoadTest
```mvn verify -Dtest=skip -Dit.test=ILoadTest* -Dscale > loadtest.txt```
```
[INFO] Running
org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
200.663 s - in org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling
[INFO] Running
org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestSessionCredentials
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
2.692 s - in
org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestSessionCredentials
[INFO] Running
org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestRoleCredentials
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
2.984 s - in org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestRoleCredentials
```
> Update AWS Java SDK to 2.27.14
> -------------------------------
>
> Key: HADOOP-19258
> URL: https://issues.apache.org/jira/browse/HADOOP-19258
> Project: Hadoop Common
> Issue Type: Task
> Components: build, fs/s3
> Reporter: Diljot Grewal
> Priority: Major
> Labels: pull-request-available
>
> Upgrade SDK to add IfNoneMatch support for upcoming PutIfNotExist
> functionality
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
