[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070617#comment-16070617
]
Steve Loughran commented on HADOOP-14553:
-----------------------------------------
Patch 004
# parallelise some more with better isolation
# Clean up some of the tests with: try with resources, use of ContracTestUtil
asserts about the state of the FS.
Metric of work, the non-scale IT tests run in under 3 minutes
{code}
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
Running
org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Running
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.643 sec - in
org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 22.088 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 22.189 sec - in
org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator
Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 1.068 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.535 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractMocked
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.207 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractMocked
Running
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.855 sec - in
org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.192 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemMocked
Running org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.493 sec - in
org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.95 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemMocked
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.926 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 30.771 sec - in
org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.445 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.468 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.282 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.534 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 94.839 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 87.913 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.172 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 106.015 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Results :
Tests run: 321, Failures: 0, Errors: 0, Skipped: 70
[INFO]
[INFO] --- maven-failsafe-plugin:2.17:integration-test
(sequential-integration-tests) @ hadoop-
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:20 min
[INFO] Finished at: 2017-06-30T20:02:21+01:00
[INFO] Final Memory: 35M/822M
[INFO] ------------------------------------------------------------------------
{code}
The big troublespot in migration and performance is the subclasses of
{{NativeAzureFileSystemBaseTest}}. These are trouble as some methods need
hardcode paths; and the overall suites are long. They are a bottleneck.
The solution here will not be to try and migrate the entire suite, but instead
split it up into sections for ease of migration and maintenance, *and* to allow
the tests to run in parallel, rather than just sequentially.
# group by function
# explicit root-path relative tests to be in their own abstract test suite,
whose implementation classes will run sequentially
# move incrementally.
> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
