[
https://issues.apache.org/jira/browse/HADOOP-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887446#comment-16887446
]
Steve Loughran commented on HADOOP-16415:
-----------------------------------------
Test run timings with HADOOP-16422 to shave 3 min off:
-Dparallel-tests -DtestsThreadCount=12 -Ds3guard -Ddynamo
Interesting to see how the parallel phase has got more tests in: the remote
file changed, S3 Select and delegation token MR jobs are all new since Hadoop
3.3.
{code}
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer
[INFO] Running
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol
[INFO] Running
org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.064 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer
[INFO] Running
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol
[WARNING] Tests run: 15, Failures: 0, Errors: 0, Skipped: 15, Time elapsed:
23.067 s - in
org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Running
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.1 s -
in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
[INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.853
s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
[INFO] Running
org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob
[WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
32.889 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
[INFO] Running org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.578 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
[INFO] Running org.apache.hadoop.fs.s3a.commit.ITestCommitOperations
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.284
s - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.147 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
[INFO] Running org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.821 s
- in org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.135 s
- in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.89 s
- in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
[INFO] Running
org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.953
s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[INFO] Running
org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.936
s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3GuardTtl
[INFO] Tests run: 72, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 108.264
s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3GuardCreate
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.302 s
- in org.apache.hadoop.fs.s3a.ITestS3GuardCreate
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91.607
s - in org.apache.hadoop.fs.s3a.commit.ITestCommitOperations
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlocksize
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
28.732 s - in org.apache.hadoop.fs.s3a.ITestS3GuardTtl
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.065 s
- in org.apache.hadoop.fs.s3a.ITestS3ABlocksize
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.711 s
- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
96.335 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
[INFO] Running org.apache.hadoop.fs.s3a.ITestAuthoritativePath
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.018
s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ACreatePerformance
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 147.394
s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.548 s
- in org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed:
10.493 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.59
s - in org.apache.hadoop.fs.s3a.scale.ITestS3ACreatePerformance
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestLocalMetadataStoreScale
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 111.437
s - in org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ADelayedFNF
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.53
s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 7.412
s - in org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AUnbuffer
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestAssumeRole
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 7.227
s - in org.apache.hadoop.fs.s3a.scale.ITestLocalMetadataStoreScale
[INFO] Running
org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed:
13.986 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
[INFO] Running
org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.79 s
- in org.apache.hadoop.fs.s3a.ITestAuthoritativePath
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.09 s -
in org.apache.hadoop.fs.s3a.ITestS3AUnbuffer
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.334 s
- in org.apache.hadoop.fs.s3a.ITestS3ADelayedFNF
[INFO] Running
org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
[INFO] Running
org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.003 s
- in org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestAssumedRoleCommitOperations
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.903 s
- in org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMetrics
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 154.871
s - in org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.65 s -
in org.apache.hadoop.fs.s3a.ITestS3AMetrics
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.606
s - in
org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AInconsistency
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.767
s - in org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AClosedFS
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.067 s
- in org.apache.hadoop.fs.s3a.ITestS3AClosedFS
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMultipartUtils
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.587 s
- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ACopyFromLocalFile
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.377 s
- in org.apache.hadoop.fs.s3a.ITestS3AMultipartUtils
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.173 s
- in org.apache.hadoop.fs.s3a.ITestS3AInconsistency
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.291
s - in org.apache.hadoop.fs.s3a.auth.ITestAssumedRoleCommitOperations
[INFO] Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
[INFO] Running org.apache.hadoop.fs.s3a.yarn.ITestS3A
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.85 s
- in org.apache.hadoop.fs.s3a.auth.ITestAssumeRole
[INFO] Running org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
[INFO] Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.672
s - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.497 s
- in org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 198.721
s - in org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3GuardWriteBack
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
15.374 s - in org.apache.hadoop.fs.s3a.ITestS3ACopyFromLocalFile
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.416 s
- in org.apache.hadoop.fs.s3a.yarn.ITestS3A
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 1.2 s
- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 236.959
s - in
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ARemoteFileChanged
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.572 s
- in org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
221.603 s - in
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.035 s
- in org.apache.hadoop.fs.s3a.ITestS3GuardWriteBack
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.323 s
- in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
26.808 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.65 s
- in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.394 s
- in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.229 s
- in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ATestUtils
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.978 s
- in org.apache.hadoop.fs.s3a.ITestS3ATestUtils
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 256.124
s - in
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3GuardEmptyDirs
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.674
s - in org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectMRJob
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.365 s
- in org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.477 s
- in org.apache.hadoop.fs.s3a.ITestS3GuardEmptyDirs
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectCLI
[WARNING] Tests run: 30, Failures: 0, Errors: 0, Skipped: 30, Time elapsed:
54.126 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3Select
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.811 s
- in org.apache.hadoop.fs.s3a.select.ITestS3SelectMRJob
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEmptyDirectory
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.635
s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectCLI
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
31.451 s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMetadataPersistenceException
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.416 s
- in org.apache.hadoop.fs.s3a.ITestS3AMetadataPersistenceException
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.29 s
- in org.apache.hadoop.fs.s3a.ITestS3AEmptyDirectory
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.028
s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 137.424
s - in org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 124.976
s - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
[INFO] Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.884
s - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 107.343
s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
[WARNING] Tests run: 73, Failures: 0, Errors: 0, Skipped: 4, Time elapsed:
308.176 s - in
org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.541
s - in org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 162.568
s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps
[INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 126.068
s - in org.apache.hadoop.fs.s3a.select.ITestS3Select
[INFO] Tests run: 55, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 170.225
s - in org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore
[INFO] Tests run: 135, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
438.568 s - in org.apache.hadoop.fs.s3a.ITestS3ARemoteFileChanged
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 1042, Failures: 0, Errors: 0, Skipped: 82
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test
(sequential-integration-tests) @ hadoop-aws ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.908 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 5.385
s - in org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits
[INFO] Running
org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed:
10.287 s - in
org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter
[INFO] Running
org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortDirectoryCommitter
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 9.031
s - in org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortDirectoryCommitter
[INFO] Running
org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.653 s
- in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 4.814
s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 4.921
s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesSSECDiskBlocks
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 4.819
s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesSSECDiskBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 4.661
s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 5, Time elapsed:
46.228 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
52.109 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale
[WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed:
7.63 s - in org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 89, Failures: 0, Errors: 0, Skipped: 67
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:18 min (Wall Clock)
[INFO] Finished at: 2019-07-17T22:35:24+01:00
[INFO] ------------------------------------------------------------------------
{code}
> Speed up S3A test runs
> ----------------------
>
> Key: HADOOP-16415
> URL: https://issues.apache.org/jira/browse/HADOOP-16415
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Priority: Major
>
> S3A Test runs are way too slow.
> Speed them by
> * reducing test setup/teardown costs
> * eliminating obsolete test cases
> * merge small tests into larger ones.
> One thing i see is that the main S3A test cases create and destroy new FS
> instances; There's both a setup and teardown cost there, but it does
> guarantee better isolation.
> Maybe if we know all test cases in a specific suite need the same options, we
> can manage that better; demand create the FS but only delete it in an
> @Afterclass method. That'd give us the OO-inheritance based setup of tests,
> but mean only one instance is done per suite
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
