[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-14553: ------------------------------------ Status: Patch Available (was: Open) unit tests locally sequentialy and with parallel at 1-2-4-12. threads = 12 triggered problems with one test. Serialized test time: 1:17; parallelised & serial, 42s, from threads >= 4. At 36s, {{TestClientThrottlingAnalyzer}} becomes the bottleneck > Add (parallelized) integration tests to hadoop-azure > ---------------------------------------------------- > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Affects Versions: 2.9.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch, > HADOOP-14553-012.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org