Thanks very much for working on this, Brett! This is a great document.
Brett Porter wrote:
Firstly - does that feature matrix look accurate to everyone involved in
this?
I do have some remarks, most of which are aimed towards improving
maven-verifier's grade.
1) This document draws a distinction between "integration testing" and
"functional testing." The two terms are defined, but I don't think I
quite understand the distinction in practice.
In particular, this doc states that only two of the available solutions
"can be used for integration testing:" the maven-it-plugin "ITP" and the
maven-verifier-plugin "VRP."
I find that puzzling, because VRP does VERY little: it just verifies that
a file exists and (optionally) that its content matches a regex. It's
hard to see how it could be used in anything but a functional test, in
which you'd run a full Maven build and then assert on the output...?
As for maven-it-plugin, I didn't think maven-it-plugin could do a
different kind of testing from what maven-invoker-plugin does... There's
even a remark to that effect in the Notes section of your wiki article:
"IT plugin is now duplicated with invoker."
What does it mean when this doc says that maven-it-plugin can do
integration testing but maven-invoker-plugin can't?
2) Three of the solutions have the feature "Runs with Surefire:"
maven-verifier (VER), maven-plugin-testing-harness (PTH), and
"maven-component-it-plugin with Java wrappers generated" (CIJ). In those
cases where they run under Surefire, I would think that a number of the
categories listed would be N/A.
For example, "able to run/timeout tests in parallel", "can ignore
failures", and "can skip execution" are pretty clearly Surefire features.
Weirdly, CIJ marks this feature as N/A, PTH gets a (/) and VER gets a (x).
But surely all three of these should have the same value, right?
For those, I think it's pretty clear that all three of them should get a
(/). But consider ANSI color support. Surefire doesn't support that
right now (I just filed it as SUREFIRE-420), so I guess those tools that
use Surefire would get a (x). But if we do wind up supporting ANSI color
some day, then they all get a (/), without ever touching the old plugins.
As for Groovy support, you can write Groovy tests and run them under
Surefire right now (I just tried it); Surefire just runs classes in
test-classes, ignorant of how they got there. Any language that compiles
to .class and extends TestCase should work, including Jython and
Rhino-compiled JavaScript. More generally, anything that runs in JUnit
will run under Surefire.
I think that means that CIJ, VER and PTH all support writing tests in
Groovy, right?
3) What is "able to run a regex set of tests?" Is that just running a
specific subset of tests defined by a regex? e.g. if you had tests
FooBar, FooBaz, and BlahBar, you could run just "Foo.*" or just ".*Bar"?
Do Surefire's wildcard-based includes count? (You know you can even run
them from the command line, with -Dtest=Foo* ...)
If not, should we introduce a new feature called "able to run a wildcard
set of tests"?
4) What is "interpolate goals/properties to run Maven with?"
5) What is "Provides alternate remote repository handling?" Apparently
nothing has it. What would it do if someone did support it?
--------------
Finally, I've got some quibbling. By my count, VER gets 39 (x)s. Of
those, I asked questions about 5 of them ("integration" testing, regexes,
interpolation of goals/properties, alternate remote repo handling), 16 I'd
let stand, and I'd like to quibble about the other 19. :-)
6) These three I think are just mistakes... they clearly work today.
* Suite teardown steps (public void tearDown)
* Cleans up logs, targets (verifier does this...?)
* Configurable working directory (that's the only argument to Verifier's
constructor!)
7) These five should all be (/), because Surefire supports them:
* Able to run tests in parallel
* Able to timeout tests run in parallel
* Can ignore failures
* Can skip execution
* Groovy support
8) VER got six (x)s that all amount to roughly the same thing: controlling
the flags that get passed to the invoked Maven, whether JVM flags or
command line arguments.
* Configurable individual Maven settings
* Pass profiles to run Maven with
* Pass profiles to run Maven with in a file
* Pass flags to run Maven with
* Pass flags to run Maven with in a file
* Able to run Maven in debug mode
But you can absolutely pass arbitrary arguments using VER, passing any
arbitrary JVM arg or command line argument. (You pass JVM args using the
MAVEN_OPTS environment variable.) Since you can pass arbitrary command
line arguments, you pass arbitrary profiles in, and can also specify
arbitrary settings.xml files with those profiles defined. And of course
you can also do it "in a file." ;-)
9) Can be used for report plugin testing: I test the surefire-report
plugin using VER; it verifies that the report ran successfully without
failing the Maven build. You can't verify a report completely without
also adding HtmlUnit (which I haven't bothered to do), but HtmlUnit is
definitely compatible with Verifier. That's at least as good as
maven-invoker-plugin, which got a (/).
10) "Isolated from user's environment/settings" and "Able to inherit
shell's environment:" In fact, Verifier inherits the shell environment by
default, which is what makes it less likely to be isolated from the user's
environment. Still, you CAN deliberately isolate yourself further, if you
wish, by specifying your environment variables when you call executeGoal,
and by passing in an explicit Maven settings file.
We should split this into "Isolated from user's environment/settings by
default" and "Can isolate from user's environment/settings." And "Able to
inherit shell environment" should be (/).
11) Can select which projects to run (include/exclude). Projects are just
resources. Tests are what you run. This should be N/A or (/); you can
certainly select which tests to run (include/exclude) using Surefire.
12) Able to drop-in an IT easily from a user submission. I don't think
this is specific enough. I think VER is pretty easy; other people
apparently disagree. This should be phrased as "able to drop in an IT
without _______" where "_______" is something hard.
That, of course, requires us to agree on what "______" should be and
whether it's hard. As I've argued in the past, some people have argued
that "_____" is "writing a JUnit test," which I insist is very easy.
It's my expectation that the outcomes would be:
1) a set of test projects that exercise different scenarios using the plugins
available today
2) document how to do so
3) identify areas of duplication, and of potential convergence
4) come up with a list of small, doable tasks that will bring this about,
updating docs as it goes
I've added one proposal to the Wiki which I don't think anybody had
objected to: make maven-verifier and maven-invoker share code.
I think if we did that, as well as improving local repository isolation in
maven-invoker (and thereby maven-verifier) would knock out most of the
remaining (x)s in maven-verifier and maven-invoker.
I'd like to ultimately tell testers: "Use PTH for unit testing, use
maven-verifier and/or maven-invoker for functional testing, and that's
it."
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
