[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128856#comment-16128856
]
Steve Loughran commented on HADOOP-14553:
-----------------------------------------
Test time overall; 13:07, with half the time coming from the
not-yet-parallelised test suites
Results :
Tests run: 319, Failures: 0, Errors: 0, Skipped: 98
[INFO]
[INFO] --- maven-failsafe-plugin:2.17:integration-test
(sequential-integration-tests) @ hadoop-azure ---
[INFO] Failsafe report directory:
/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/failsafe-reports
{code}
Running org.apache.hadoop.fs.azure.ITestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.982 sec - in
org.apache.hadoop.fs.azure.ITestBlobDataValidation
Running org.apache.hadoop.fs.azure.ITestBlockBlobInputStream
Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed: 0.989 sec -
in org.apache.hadoop.fs.azure.ITestBlockBlobInputStream
Running
org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.407 sec -
in
org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 228.507 sec -
in org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads
Running
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAuthorizationWithOwner
Tests run: 27, Failures: 0, Errors: 0, Skipped: 27, Time elapsed: 3.862 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAuthorizationWithOwner
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.435 sec - in
org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 274.746 sec -
in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.224 sec - in
org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive
Running org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 1.32 sec - in
org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper
{code}
> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
