[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061505#comment-16061505
]
Steve Loughran commented on HADOOP-14553:
-----------------------------------------
patch 002
There's some careful parallelization of a few tests, enough to show that it
does work if you do it case by case, the use of shared/hard coded paths being
the enemy. I've moved over all the contract tests as we know they parallize
well once you set up the contract to return fork-specific paths.
I've uprated a limited set oftests so far, enough to show things work and to
achieve measurable speeups already. Here is a test where the total time to run
these tests is 3 minutes, rather than the 5 minutes it would take otherwise.
Tagging the huge files and read and seek as scale tests means that non-scale
test runs would be even faster.
Moved:
* all the contract tests in {{org.apache.hadoop.fs.azure.contract}}, tweaking
{{NativeAzureFileSystemContract}} to use the fork ID sysprop to generate unique
paths.
* {{ITestFileSystemOperationExceptionMessage}}. This was taking so long as it
was needlessly subclassing {{NativeAzureFileSystemBaseTest}} and its 50 tests.
Moving the single method to a new IT test fixed that.
* {{ITestNativeAzureFileSystemContractLive}}: again, known to parallelize.
* {{ITestReadAndSeekPageBlobAfterWrite}} self-contained, slow (it's been tagged
as a scale test), and validates the ability for pageblob paths to be per-fork
too.
Added
* docs in `test_azure.md`
{code}
> mvit -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dtest=none
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 11.567 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.605 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.765 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Running
org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.948 sec - in
org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.087 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Running org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.702 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.769 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.872 sec - in
org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.163 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 79.653 sec -
in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 98.776 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 150.19 sec - in
org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 165.022 sec -
in org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Results :
Tests run: 138, Failures: 0, Errors: 0, Skipped: 6
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:14 min (Wall Clock)
[INFO] Finished at: 2017-06-23T21:52:01+01:00
[INFO] Final Memory: 38M/820M
[INFO] ------------------------------------------------------------------------
{code}
You need to run the full tests with `mvn verify`, for some reason they don't
run on `mvn integration-test`, whereas they do on hadoop-aws.
The main tests still take forever, for a loose definition of forever, they
still leak wasb storage containers. As well as tracking that down, I'm tempted
to have an IT test in the sequential phase which lists all matching containers
and deletes them. For now, main issue is making sure things don't time out. Of
course, the more that get moved to parallel tests, the faster the runs, the
less risk of timeouts.
I don't plan to move any more tests over in this JIRA: there's enough to set up
the parallel test execution framework & explore its functionality, with the new
seek test it's set up for seek changes. The remaining tests would go if/when
{{AbstractWasbTestBase}} was brought up as an IT tests, then case-by-case as
paths get isolated. The slow tests like {{TestAzureFileSystemInstrumentation}}
would be the first targets.
Also, I'd look really hard about whether it's worthwhile supporting the
emulation. Life is simpler if the policy is just "test against real Azure".
Something for another day, but people should think about it. A test run doesn't
cost much and its more realistic; even long-haul test runs like mine get to
find interesting network errors.
Would be good for others to play with this, perhaps with some review by
[~cnauroth].
> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
