[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070670#comment-16070670
]
Steve Loughran commented on HADOOP-14553:
-----------------------------------------
Tested, Azure ireland. Test run time now down to 23 minutes, with some obvious
tests taking 1-5 minutes each: parallelise them and most of the heavy lifting
is done. But two of those are the native tests: metadata heavy, but a long set
of operations
{code}
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 96.646 sec - in
org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385 sec - in
org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.829 sec - in
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.234 sec - in
org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.482 sec - in
org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Running org.apache.hadoop.fs.azure.TestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.656 sec - in
org.apache.hadoop.fs.azure.TestBlobDataValidation
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.83 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.547 sec - in
org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Running org.apache.hadoop.fs.azure.TestContainerChecks
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.368 sec - in
org.apache.hadoop.fs.azure.TestContainerChecks
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.276 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Running
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.692 sec -
in
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 416.005 sec -
in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.004 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed: 3.397 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Tests run: 24, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 4.036 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.884 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.062 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.883 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.862 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 355.53 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.456 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.107 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 284.874 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.814 sec - in
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.126 sec - in
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.099 sec - in
org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Running org.apache.hadoop.fs.azure.TestWasbFsck
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.709 sec - in
org.apache.hadoop.fs.azure.TestWasbFsck
Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 1.791 sec - in
org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Tests run: 18, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 7.907 sec - in
org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Results :
Tests run: 341, Failures: 0, Errors: 0, Skipped: 60
[INFO]
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-azure ---
[INFO] Building jar:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/hadoop-azure-3.0.0-beta1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-azure ---
[INFO] Building jar:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/hadoop-azure-3.0.0-beta1-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @
hadoop-azure ---
[INFO]
[INFO] --- maven-failsafe-plugin:2.17:integration-test
(default-integration-test) @ hadoop-azure ---
[INFO] Failsafe report directory:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/failsafe-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
Running
org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Running
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.914 sec - in
org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 9.681 sec - in
org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.059 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator
Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.516 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 20.332 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractMocked
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 1.589 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractMocked
Running
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.072 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemMocked
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.169 sec - in
org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.026 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 8.103 sec - in
org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.812 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.191 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemMocked
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.306 sec - in
org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.433 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.239 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.019 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.856 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.025 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 90.316 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 87.942 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive
Results :
Tests run: 321, Failures: 0, Errors: 0, Skipped: 70
[INFO]
[INFO] --- maven-failsafe-plugin:2.17:integration-test
(sequential-integration-tests) @ hadoop-azure ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (depcheck) @ hadoop-azure ---
[INFO]
[INFO] --- maven-failsafe-plugin:2.17:verify (default-integration-test) @
hadoop-azure ---
[INFO] Failsafe report directory:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/failsafe-reports
[INFO]
[INFO] --- maven-failsafe-plugin:2.17:verify (sequential-integration-tests) @
hadoop-azure ---
[INFO] Failsafe report directory:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/failsafe-reports
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:13 min
[INFO] Finished at: 2017-06-30T20:58:23+01:00
[INFO] Final Memory: 46M/834M
[INFO] ------------------------------------------------------------------------
{code}
> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
