[
https://issues.apache.org/jira/browse/LUCENE-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Weiss resolved LUCENE-3492.
---------------------------------
Resolution: Fixed
Fix Version/s: (was: 4.0)
I consider this issue done. I've pulled all the super-cool features from
Lucene/Solr and put them into a separate, stand-alone and reusable project
called randomizedtesting. We have switched our infrastructure at Carrot2 to use
it and I'm very happy with it.
http://labs.carrotsearch.com/randomizedtesting.html
I will file another issue concerned with progressively moving from
LuceneTestRunner/Case and tests infrastructure to RandomizedRunner (and
siblings).
> Extract a generic framework for running randomized tests.
> ---------------------------------------------------------
>
> Key: LUCENE-3492
> URL: https://issues.apache.org/jira/browse/LUCENE-3492
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/test
> Reporter: Dawid Weiss
> Assignee: Dawid Weiss
> Priority: Minor
> Attachments: Screen Shot 2011-10-06 at 12.58.02 PM.png
>
>
> {color:red}The work on this issue is temporarily at github{color} (lots of
> experiments and tweaking):
> https://github.com/carrotsearch/randomizedtesting
> Or directly: git clone git://github.com/carrotsearch/randomizedtesting.git
> {color}
> ----
> RandomizedRunner is a JUnit runner, so it is capable of running
> @Test-annotated test cases. It
> respects regular lifecycle hooks such as @Before, @After, @BeforeClass or
> @AfterClass, but it
> also adds the following:
> Randomized, but repeatable execution and infrastructure for dealing with
> randomness:
> - uses pseudo-randomness (so that a given run can be repeated if given the
> same starting seed)
> for many things called "random" below,
> - randomly shuffles test methods to ensure they don't depend on each other,
> - randomly shuffles hooks (within a given class) to ensure they don't depend
> on each other,
> - base class RandomizedTest provides a number of methods for generating
> random numbers, strings
> and picking random objects from collections (again, this is fully
> repeatable given the
> initial seed if there are no race conditions),
> - the runner provides infrastructure to augment stack traces with information
> about the initial
> seeds used for running the test, so that it can be repeated (or it can be
> determined that
> the test is not repeatable -- this indicates a problem with the test case
> itself).
> Thread control:
> - any threads created as part of a test case are assigned the same initial
> random seed
> (repeatability),
> - tracks and attempts to terminate any Threads that are created and not
> terminated inside
> a test case (not cleaning up causes a test failure),
> - tracks and attempts to terminate test cases that run for too long (default
> timeout: 60 seconds,
> adjustable using global property or annotations),
> Improved validation and lifecycle support:
> - RandomizedRunner uses relaxed contracts of hook methods' accessibility
> (hook methods _can_ be
> private). This helps in avoiding problems with method shadowing (static
> hooks) or overrides
> that require tedious super.() chaining). Private hooks are always executed
> and don't affect
> subclasses in any way, period.
> - @Listeners annotation on a test class allows it to hook into the execution
> progress and listen
> to events.
> - @Validators annotation allows a test class to provide custom validation
> strategies
> (project-specific). For example a base class can request specific test case
> naming strategy
> (or reject JUnit3-like methods, for instance).
> - RandomizedRunner does not "chain" or "suppress" exceptions happening during
> execution of
> a test case (including hooks). All exceptions are reported as soon as they
> happened and multiple
> failure reports can occur. Most environments we know of then display these
> failures sequentially
> allowing a clearer understanding of what actually happened first.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
