diljotgrewal commented on PR #7015:
URL: https://github.com/apache/hadoop/pull/7015#issuecomment-2327918461
ran the scale tests with and without sdk update commit:
#### without AWS SDK update:
```
[ERROR]
testCostOfCreatingMagicFile(org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost)
Time elapsed: 2.922 s <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0004/test/testCostOfCreatingMagicFile/__magic_job-123/__base/file.txt.pending':
Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException:
Unexpected end-of-input: expected close marker for Object (start marker at
[Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 640])
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column:
3629] (through reference chain:
org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit["iostatistics"]):
Unexpected end-of-input: expected close marker for Object (start marker at
[Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 640])
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column:
3629] (through reference chain:
org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit["iostatistics"])
```
```
[ERROR]
test_200_execute[magic](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob)
Time elapsed: 37.374 s <<< FAILURE!
java.lang.AssertionError:
Job job_1725412925883_0003 failed in state FAILED with cause Job commit
failed: org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0008/test/ITestS3ACommitterMRJob-execute-magic/__magic_job-3cbf58ed-920a-48a7-9a68-2b888fb9ce11/job-3cbf58ed-920a-48a7-9a68-2b888fb9ce11/01/task_1725412925883_0003_m_000000.pendingset':
Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException:
Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column:
4270] (through reference chain:
org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]):
Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column:
4270] (through reference chain:
org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])
```
```
[ERROR] Tests run: 5, Failures: 4, Errors: 0, Skipped: 0, Time elapsed:
20.563 s <<< FAILURE! - in
org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms
[ERROR]
testEncryptionOverRename(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)
Time elapsed: 5.037 s <<< FAILURE!
java.lang.AssertionError: Key wrap algo isn't same as expected
expected:<kms+context> but was:<null>
```
```
[ERROR]
testDirectoryListingFileLengths(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)
Time elapsed: 3.812 s <<< FAILURE!
java.lang.AssertionError:
[File lengths aren't the same as expected from FileStatus dir. listing]
Expecting:
<[0, 1, 239, 4079]>
to contain exactly in any order:
<[0, 1, 255, 4095]>
elements not found:
<[255, 4095]>
and elements not expected:
<[239, 4079]>
```
```
[ERROR]
testEncryptionEnabledAndDisabledFS(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)
Time elapsed: 1.837 s <<< FAILURE!
java.lang.AssertionError: SecurityException should be thrown: "Exception
should be raised if unencrypted data is read by a CSE enabled FS"
```
```
[ERROR] Tests run: 18, Failures: 0, Errors: 10, Skipped: 1, Time elapsed:
21.693 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion
[ERROR]
testCentralEndpointAndNullRegionFipsWithCRUD(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion)
Time elapsed: 2.665 s <<< ERROR!
java.lang.IllegalArgumentException: Non central endpoint cannot be set when
fs.s3a.endpoint.fips is true : https://s3-accesspoint.us-east-1.amazonaws.com
```
```
[ERROR] testCentralEndpoint(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion)
Time elapsed: 0.948 s <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Invalid
configuration: region from ARN `us-east-1` does not match client region
`us-east-2` and UseArnRegion is `false`
```
```
[^[[1;31mERROR^[[m]
testEndpointOverride(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion) Time
elapsed: 0.998 s <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Invalid
configuration: region from ARN `us-east-1` does not match client region
`us-east-2` and UseArnRegion is `false`
```
```
[ERROR]
testCommitterWithDuplicatedCommit[track-commit-in-memory-false](org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol)
Time elapsed: 9.857 s <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0002/test/ITestMagicCommitProtocol-testCommitterWithDuplicatedCommit[track-commit-in-memory-false]/__magic_job-job_202409040534_0002/job-job_202409040534_0002/01/task_202409040534_0002_m_000000.pendingset':
Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException:
Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column:
4331] (through reference chain:
org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]):
Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column:
4331] (through reference chain:
org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])
```
```
[ERROR]
testMapFileOutputCommitter[track-commit-in-memory-false](org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol)
Time elapsed: 11.016 s <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0002/test/ITestMagicCommitProtocol-testMapFileOutputCommitter[track-commit-in-memory-false]/__magic_job-job_202409040503_0002/job-job_202409040503_0002/01/task_202409040503_0002_m_000000.pendingset':
Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException:
Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column:
5604] (through reference chain:
org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]):
Unexpected end-of-input in field name
at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column:
5604] (through reference chain:
org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])
```
```
[ERROR]
testComplexDirActions(org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List)
Time elapsed: 16.843 s <<< FAILURE!
java.lang.AssertionError: getLen expected:<512> but was:<496>
```
```
[ERROR]
testRenamePopulatesFileAncestors2(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)
Time elapsed: 4.976 s <<< FAILURE!
java.lang.AssertionError: Number of bytes copied: files_copied_bytes
expected:<16384> but was:<16368>
```
```
[ERROR]
testLargeFilesFromRemote(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp)
Time elapsed: 5.72 s <<< ERROR!
java.io.FileNotFoundException: File
file:/home/dsgrewal/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/ITestS3AContractDistCp/testLargeFilesFromRemote/local/outputDir/inputDir/file1
does not exist
```
```
[ERROR]
test_110_teragen[magic-memory=false](org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A)
Time elapsed: 20.265 s <<< FAILURE!
java.lang.AssertionError: teragen(1000,
s3a://s3atestingdsgrewal/job-00/test/terasort-magic-false/sortin) failed
expected:<0> but was:<1>
```
```
[ERROR]
testStatistics(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics)
Time elapsed: 2.27 s <<< FAILURE!
java.lang.AssertionError: Mismatch in bytes written expected:<697> but
was:<512>
```
```
[ERROR]
test_030_postCreationAssertions(org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesStorageClass)
Time elapsed: 0.572 s <<< FAILURE!
org.junit.ComparisonFailure: [List file status length
S3AFileStatus{path=s3a://s3atestingdsgrewal/job-00/test/tests3ascale/array/src/hugefile;
isDirectory=false; length=10485744; replication=1; blocksize=33554432;
modification_time=1725414230000; access_time=0; owner=dsgrewal; group=dsgrewal;
permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true;
isErasureCoded=false} isEmptyDirectory=FALSE
eTag="8f247ca05cb4324d2bd3e1ec1a88e481-2" versionId=null]
expected:<104857[60]L> but was:<104857[44]L>
```
#### With SDK update
command : ``
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
