[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128856#comment-16128856 ]
Steve Loughran commented on HADOOP-14553: ----------------------------------------- Test time overall; 13:07, with half the time coming from the not-yet-parallelised test suites Results : Tests run: 319, Failures: 0, Errors: 0, Skipped: 98 [INFO] [INFO] --- maven-failsafe-plugin:2.17:integration-test (sequential-integration-tests) @ hadoop-azure --- [INFO] Failsafe report directory: /Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/failsafe-reports {code} Running org.apache.hadoop.fs.azure.ITestBlobDataValidation Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.982 sec - in org.apache.hadoop.fs.azure.ITestBlobDataValidation Running org.apache.hadoop.fs.azure.ITestBlockBlobInputStream Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed: 0.989 sec - in org.apache.hadoop.fs.azure.ITestBlockBlobInputStream Running org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.407 sec - in org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded Running org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 228.507 sec - in org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAuthorizationWithOwner Tests run: 27, Failures: 0, Errors: 0, Skipped: 27, Time elapsed: 3.862 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAuthorizationWithOwner Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.435 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 274.746 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive Running org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.224 sec - in org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive Running org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 1.32 sec - in org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper {code} > Add (parallelized) integration tests to hadoop-azure > ---------------------------------------------------- > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Affects Versions: 2.9.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org