Steve Loughran updated HADOOP-14553:
    Status: Patch Available  (was: Open)

> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>                 Key: HADOOP-14553
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14553
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, 
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, 
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, 
> HADOOP-14553-009.patch, HADOOP-14553-010.patch
> The Azure tests are slow to run as they are serialized, as they are all 
> called Test* there's no clear differentiation from unit tests which Jenkins 
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize 
> (which includes having separate paths for every test suite). The code in 
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*:  Adding a list of requirements to clarify the 
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) 
> IntegrationTests - requires Azure Storage account, executed by engineers 
> prior to check-in, and if needed, iii) ScaleTests – long running performance 
> and scalability tests.
> # Define configuration profiles to run tests with different settings.  Allows 
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and 
> false.  Need to review settings to see what else would benefit.
> # Maven commands to run b) and c).  Turns out it is not easy to do with 
> Maven, so we might have to run it multiple times to run with different 
> configuration settings.
> # Document how to add and run tests and the process for contributing to 
> Apache Hadoop.  Steve shared an example at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
> # UnitTests should run in under 2 minutes and IntegrationTests should run in 
> under 15 minutes, even on slower network connections.  (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted.  Exceptions for large 
> persistent content used repeatedly to expedite test execution. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to