[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marquardt updated HADOOP-14553:
--------------------------------------
Description:
The Azure tests are slow to run as they are serialized, as they are all called
Test* there's no clear differentiation from unit tests which Jenkins can run,
and integration tests which it can't.
Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
(which includes having separate paths for every test suite). The code in
hadoop-aws's POM &c show what to do.
*UPDATE August 4, 2017*: Adding a list of requirements to clarify the
acceptance criteria for this JIRA:
# Parallelize test execution
# Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
IntegrationTests - requires Azure Storage account, executed by engineers prior
to check-in, and if needed, iii) ScaleTests – long running performance and
scalability tests.
# Define configuration profiles to run tests with different settings. Allows
an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
false. Need to review settings to see what else would benefit.
# Maven commands to run b) and c). Turns out it is not easy to do with Maven,
so we might have to run it multiple times to run with different configuration
settings.
# Document how to add and run tests and the process for contributing to Apache
Hadoop. Steve shared an example at
https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
# UnitTests should run in under 2 minutes and IntegrationTests should run in
under 15 minutes, even on slower network connections. (These are rough goals)
# Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
persistent content used repeatedly to expedite test execution.
was:
The Azure tests are slow to run as they are serialized, as they are all called
Test* there's no clear differentiation from unit tests which Jenkins can run,
and integration tests which it can't.
Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
(which includes having separate paths for every test suite). The code in
hadoop-aws's POM &c show what to do.
> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
