[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14553:
------------------------------------
Status: Patch Available (was: Open)
Tested: Azure ireland.
Test run without scale tests complete in just under 13 minutes; with scale: 21.
That's better than before and with scope for speedup, as much of the test time
is those IT tests not running in the parallel phase
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.353 sec - in
org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.753 sec - in
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.222 sec - in
org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.798 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.211 sec - in
org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
Running org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.787 sec - in
org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
Tests run: 24, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 4.568 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.87 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.04 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 1.315 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.81 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.246 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.424 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.069 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.851 sec - in
org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.107 sec - in
org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Running org.apache.hadoop.fs.azure.TestWasbFsck
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.712 sec - in
org.apache.hadoop.fs.azure.TestWasbFsck
Results :
Tests run: 214, Failures: 0, Errors: 0, Skipped: 35
[INFO]
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-azure ---
[INFO] Building jar:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/hadoop-azure-3.0.0-beta1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-azure ---
[INFO] Building jar:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/hadoop-azure-3.0.0-beta1-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @
hadoop-azure ---
[INFO]
[INFO] --- maven-failsafe-plugin:2.17:integration-test
(default-integration-test) @ hadoop-azure ---
[INFO] Failsafe report directory:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/failsafe-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Running
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.358 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
Tests run: 7, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 20.871 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.1 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.ITestAzureFileSystemErrorConditions
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.861 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.ITestBlobTypeSpeedDifference
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.299 sec - in
org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Running org.apache.hadoop.fs.azure.ITestContainerChecks
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.66 sec - in
org.apache.hadoop.fs.azure.ITestBlobTypeSpeedDifference
Running org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionHandling
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.379 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionMessage
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.744 sec - in
org.apache.hadoop.fs.azure.ITestContainerChecks
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.545 sec - in
org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.739 sec - in
org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionMessage
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.876 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator
Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.597 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.683 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.905 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Running
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive
Running org.apache.hadoop.fs.azure.ITestNativeAzureFSAuthorizationCaching
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.484 sec - in
org.apache.hadoop.fs.azure.ITestAzureFileSystemErrorConditions
Running org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Tests run: 28, Failures: 0, Errors: 0, Skipped: 28, Time elapsed: 7.289 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFSAuthorizationCaching
Running org.apache.hadoop.fs.azure.ITestWasbUriAndConfiguration
Tests run: 19, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 10.466 sec -
in org.apache.hadoop.fs.azure.ITestWasbUriAndConfiguration
Running org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.519 sec - in
org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.609 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.474 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.612 sec -
in org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionHandling
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 125.604 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 130.453 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 131.042 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 132.045 sec -
in org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 209.009 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 319.551 sec -
in org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Results :
Tests run: 319, Failures: 0, Errors: 0, Skipped: 83
[INFO]
[INFO] --- maven-failsafe-plugin:2.17:integration-test
(sequential-integration-tests) @ hadoop-azure ---
[INFO] Failsafe report directory:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/failsafe-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.ITestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.776 sec - in
org.apache.hadoop.fs.azure.ITestBlobDataValidation
Running org.apache.hadoop.fs.azure.ITestBlockBlobInputStream
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 225.783 sec -
in org.apache.hadoop.fs.azure.ITestBlockBlobInputStream
Running
org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.2 sec - in
org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 224.11 sec -
in org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads
Running
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAuthorizationWithOwner
Tests run: 27, Failures: 0, Errors: 0, Skipped: 27, Time elapsed: 4.248 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAuthorizationWithOwner
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.728 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 301.692 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.833 sec - in
org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive
Running org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 1.803 sec -
in org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper
Results :
Tests run: 153, Failures: 0, Errors: 0, Skipped: 37
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20:41 min (Wall Clock)
[INFO] Finished at: 2017-08-03T16:30:02+01:00
[INFO] Final Memory: 42M/837M
[INFO] ------------------------------------------------------------------------
{code}
> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
