[
https://issues.apache.org/jira/browse/HADOOP-15664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599725#comment-16599725
]
Thomas Marquardt commented on HADOOP-15664:
-------------------------------------------
This is a great improvement to test execution time! Thanks Da!
A couple tests failed for me so I made the following changes and uploaded 003
patch:
# Increased the last modified time error margin from 10 to 60 seconds in
*NativeAzureFileSystemBaseTest.java*. There is frequently clock skew when
comparing server and client clocks, so I have increased the error margin. I
have seen the tests that rely on this fail from time to time, so this will help
improve robustness.
# The ITestAzureBlobFileSystemFlush.*testFlushWithFlushEnabled*/Disabled tests
both failed. These tests were overly complex for what they needed to do, and
are now more simple. They correctly validate AbfsOutputStream.flush behavior
when fs.azure.enable.flush is set to true or false.
For me, test run time was reduced from approximately 1 hour to run both WASB
and ABFS tests to 20 minutes to run WASB plus 4 minutes to run ABFS. Note that
the improvements were only made to the new parallel-tests-wasb and
parallel-tests-abfs profiles, and the parallel-tests profile which runs all the
tests was not modified. In the future we may be able to improve parallel-tests
too, but there is currently an issue with running the contract tests
concurrently.
My storage account is on the east coast of USA and the tests were run from the
west coast of USA. Tests run extremely fast when the client and server are in
the same data center.
*mvn -T 1C -Pparallel-tests -Dscale -DtestsThreadCount=8 clean verify*
Tests run: 265, Failures: 0, Errors: 0, Skipped: 11
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
Tests run: 862, Failures: 0, Errors: 0, Skipped: 264
Tests run: 186, Failures: 0, Errors: 0, Skipped: 10
*Total time: 01:04 h (Wall Clock)*
*Tests Successfully Run = 1029*
***mvn -T 1C -Pparallel-tests-wasb -Dscale -DtestsThreadCount=8 clean verify*
Tests run: 240, Failures: 0, Errors: 0, Skipped: 11
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
Tests run: 619, Failures: 0, Errors: 0, Skipped: 65
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
*Total time: 20:04 min (Wall Clock)*
*Tests Successfully Run = 787*
*mvn -T 1C -Pparallel-tests-abfs -Dscale -DtestsThreadCount=8 clean verify*
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0
Tests run: 260, Failures: 0, Errors: 0, Skipped: 182
Tests run: 167, Failures: 0, Errors: 0, Skipped: 27
*Total time: 03:28 min (Wall Clock)*
*Tests Successfully Run = 242*
> ABFS: Reduce test run time via parallelization and grouping
> -----------------------------------------------------------
>
> Key: HADOOP-15664
> URL: https://issues.apache.org/jira/browse/HADOOP-15664
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Reporter: Thomas Marquardt
> Assignee: Da Zhou
> Priority: Major
> Attachments: HADOOP-15664-HADOOP-15407-001.patch,
> HADOOP-15664-HADOOP-15407-002.patch, HADOOP-15664-HADOOP-15407-003.patch
>
>
> 1) Let's reduce the total test runtime by improving parallelization of the
> tests.
> 2) Let's make it possible to select WASB tests, ABFS tests, or both so
> developers can run only the tests appropriate for the change they've made.
> 3) Update the testing-azure.md accordingly
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
