[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119960#comment-16119960
]
Steve Loughran commented on HADOOP-14553:
-----------------------------------------
The next iteration of this will include a test class which must be explicitly
invoked on the command line via a -Dtest=, which will list and delete all
{{wasbtests-}} containers of an account. This can be used to clean up container
leakage:
{code}
2017-08-09 15:08:54,335 INFO [JUnit-testDeleteContainers]:
azure.AbstractWasbTestBase
(CleanupTestContainers.java:testDeleteContainers(79)) - Container
wasbtests-stevel-1501782115769 URI
http://contender.blob.core.windows.net/wasbtests-stevel-1501782115769
2017-08-09 15:08:54,390 INFO [JUnit-testDeleteContainers]:
azure.AbstractWasbTestBase
(CleanupTestContainers.java:testDeleteContainers(79)) - Container
wasbtests-stevel-1501782117324 URI
http://contender.blob.core.windows.net/wasbtests-stevel-1501782117324
2017-08-09 15:08:54,444 INFO [JUnit-testDeleteContainers]:
azure.AbstractWasbTestBase
(CleanupTestContainers.java:testDeleteContainers(79)) - Container
wasbtests-stevel-1501782149411 URI
http://contender.blob.core.windows.net/wasbtests-stevel-1501782149411
2017-08-09 15:08:54,497 INFO [JUnit-testDeleteContainers]:
azure.AbstractWasbTestBase
(CleanupTestContainers.java:testDeleteContainers(86)) - Deleted 2436 test
containers
{code}
> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
