[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128882#comment-16128882
]
Steve Loughran edited comment on HADOOP-14553 at 8/16/17 2:46 PM:
------------------------------------------------------------------
Patch 012:
parallel execution of unit test except for one test
{{TestRollingWindowAverage}} which becomes unreliable in highly parallel runs
if it oversleeps
{code}
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.481 sec <<<
FAILURE! - in org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
testBasicFunctionality(org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage)
Time elapsed: 0.152 sec <<< FAILURE!
java.lang.AssertionError: expected:<10> but was:<15>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage.testBasicFunctionality(TestRollingWindowAverage.java:36)
{code}
was (Author: [email protected]):
Patch 012:
parallel execution of unit test except for one test
{{TestRollingWindowAverage}} which becomes unreliable in highly parallel runs
if it oversleeps
> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch,
> HADOOP-14553-012.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
