[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14553:
------------------------------------
Attachment: HADOOP-14553-007.patch
patch 007; some minor tuning of tests to address a couple of (transient)
failures.
Testing: Azure ireland. Total ITest execution with 12 threads; 13 minutes, down
from 30. Which is good enough for now). The parallel phase is <3 minutes, rest
is the serialized set.
There's a couple of tests where its obvious that parallelisation would deliver
the most speedup, {{ITestNativeAzureFileSystemLive}} and
{{.ITestFileSystemOperationsWithThreads}}. Those are about nine minutes between
them.
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Running
org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.75 sec - in
org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 9.974 sec - in
org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Running org.apache.hadoop.fs.azure.ITestAzureFileSystemErrorConditions
Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 17.493 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.ITestBlobTypeSpeedDifference
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.138 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.ITestContainerChecks
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.49 sec - in
org.apache.hadoop.fs.azure.ITestAzureFileSystemErrorConditions
Running org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionHandling
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.7 sec - in
org.apache.hadoop.fs.azure.ITestContainerChecks
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.383 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAtomicRenameDirList
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.986 sec - in
org.apache.hadoop.fs.azure.ITestBlobTypeSpeedDifference
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.214 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator
Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.761 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.617 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Running
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.985 sec - in
org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.768 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.016 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.642 sec - in
org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 11.966 sec - in
org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.452 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.307 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.829 sec -
in org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionHandling
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 115.612 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 132.824 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 127.923 sec -
in org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 137.281 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 139.442 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 182.182 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp
Results :
Tests run: 270, Failures: 0, Errors: 0, Skipped: 65
[INFO]
[INFO] --- maven-failsafe-plugin:2.17:integration-test
(sequential-integration-tests) @ hadoop-azure ---
[INFO] Failsafe report directory:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/failsafe-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.ITestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.746 sec - in
org.apache.hadoop.fs.azure.ITestBlobDataValidation
Running
org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.84 sec - in
org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 237.739 sec -
in org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads
Running
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAuthorizationWithOwner
Tests run: 24, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 2.464 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAuthorizationWithOwner
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.193 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 285.22 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.936 sec - in
org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive
Running org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 1.075 sec -
in org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper
Running org.apache.hadoop.fs.azure.ITestWasbUriAndConfiguration
Tests run: 18, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 7.319 sec - in
org.apache.hadoop.fs.azure.ITestWasbUriAndConfiguration
Results :
Tests run: 148, Failures: 0, Errors: 0, Skipped: 35
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (depcheck) @ hadoop-azure ---
[INFO]
[INFO] --- maven-failsafe-plugin:2.17:verify (default-integration-test) @
hadoop-azure ---
[INFO] Failsafe report directory:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/failsafe-reports
[INFO]
[INFO] --- maven-failsafe-plugin:2.17:verify (sequential-integration-tests) @
hadoop-azure ---
[INFO] Failsafe report directory:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/failsafe-reports
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:10 min
[INFO] Finished at: 2017-07-07T14:22:47+01:00
[INFO] Final Memory: 34M/818M
[INFO] ------------------------------------------------------------------------
{code}
> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
