I think what you're advocating is much of what one would achieve with per-provider plugins. The idea here is to extract the common functionality of surefire (some of which is *hard* to get right) into separate module(s) which each plugin can choose to use or not. To achieve a certain kind of end-user consistency in how the user specifies parameters they can choose to use the "common properties" interfaces to define the mojo attributes.
The net long term effect is that you'd probably end up with a number of different testng providers, one less-is-more plugin or a "surefire traditional testng" plugin and another "more is more" plugin that would support the whole shebang and then some. And the ones that support forking execution would *probably* be using the surefire forking module, but the per-provider plugin strategy mandates very few such absolute choices, since the different mojos can choose which bits to use free of constraints from what the others are doing. Personally I will not implement any solution that does not have a decent migration path for existing users, so any option that alienates existing users is a clear -1 from me. Part of the problem with existing surefire is that it's overly prescriptive in terms of what must be shared/common between the different providers. And quite a few of these assumptions are just not generalizable; they turned out to be relevant for only 1 provider. So I can easily see this turning into a small handful of old-skool providers here at apache, with various forks popping up for less-is-more or more-is-more at github. I kind of like that kind of ecosystem, which would give a freedom of choice that is not available currently. Kristian fr., 19.11.2010 kl. 12.13 -0800, skrev Rex Hoffman: > I have a slight concern. > > It goes into testing portability (running in maven, or eclipse, idea, > netbeans, or via command line) > In short, the smarter we make maven about running tests, the less easy it is > for a developer to run that same test via IDE or something similar. > > I'll give two common examples.... and hope it makes the point. > > If we add test-ng listeners to the maven surefire plugin config, those same > tests wont run the same in eclipse without adding the listeners as > annotations to the class. Eliminating the need to add the Listeners to > maven config in the first place. > > Same is true of the life cycle phases around integration tests. > > People often start jetty and run selenium style test in the integration test > phase via maven. It might be better to use an annotations or something > similar on that tests that loads the needed data and starts the application > (if not already running). That way a right click and a "run as" in an IDE > would be enough to run the test. > > Seems like energy is being put into allowing and promoting sub-optimal > solutions. Maybe helper libraries meant to be used by test developers, that > are not tied to maven, but make the running of tests in maven cleaner would > be a better approach to promote? Move the mountain as it were.... > > So in short maybe a C approach. > > Keep maven test running as simple and unconfigurable as possible, provide > some libraries and documentation to bridge any gaps. Forking config, > sure... Ant style file includes, definitely, anything that has to do > specifically with the configuration of any implementation of a test runner > should not possible via maven. > > Not sure if this is useful with what has been already implemented. But > while getting companies to switch to maven and start testing, it has proven > a challenge in the past, and this solution has been the one I found that > worked for devs, qa and automation teams. > > I wrote a small library for testNG: > > http://www.epolity.net/blog/2010/04/testng-extensions-to-bridge-the-small-gap-between-maven-and-testng/ > > > and found that the less config I put in maven, the easier it was to develop > and run tests. > > My two pennies.... > > Rex > > On Fri, Nov 19, 2010 at 10:58 AM, Kristian Rosenvold < > kristian.rosenv...@gmail.com> wrote: > > > I have been refactoring quite heavily on the surefire plugin the last > > weeks. This has been entirely non-functional with the intent of > > loosening the internal dependencies (without breaking the existing > > plugin) enough to create a starting point for solving SUREFIRE-141. > > > > At the moment I am quite satisfied with > > where the plugin is at, and we need to take some decisions on how > > the SUREFIRE-141 can be solved. Those of you who are familiar with the > > "old" surefire might want to have a look at svn HEAD, since there's been > > considerable changes around the booter. At the moment I feel this > > discussion is required before I can move on: > > > > I have two different proposals here. Both of them have some details that > > need to be solved, but I have come to the conclusion that the key issue > > to be resolved is how parameters arrive at the individual providers: > > > > A) "Pile it up" > > > > Adding all the provider-specific settings to the main surefire plugin > > was probably a mistake; but this solution assumes we'll have to live > > with that. The current list of parameters at > > http://maven.apache.org/plugins/maven-surefire-plugin/test-mojo.html is > > probably reasonably close to complete for java xUnit, leaving room for > > some future expansion there might be 5-10 more for java, and maybe (wild > > guess) 20-30 other parameters for other languages, assuming they re-use > > the existing parameters for similar meanings. All in all it could be > > handled by documentation. > > - Technically we could just transparently serialize ANY parameter given > > to the surefire-plugin through the booter to the provider, without the > > surefire plugin trying to find out if the provider will use them or not. > > The provider knows its own requirements and will sort it out. > > - Any new parameters required by a new provider would have to be added > > to the surefire-plugin. > > - Detect user-specified providers in the suerfire-plugin's > > dependencies, which would disable the current autodetection. > > - The only real piece of work remaining for this is cleaning up the > > provider instantiation & parameter transfer. Could/would probably be > > feature complete with 2.7 release. > > > > B) Per-provider plugins > > Surefire is basically a library providing classpath scanning, forking > > and reporting services. Extract the necessary interfaces so that a > > plugin can simply declare a dependency on the required services and have > > them wired in via plexus or similar. > > - Extract interfaces so that similar features (e.g. fork configuration, > > classloader settings and reporting settings) become homogenous in terms > > of plugin parameters, even though there is one distinct mojo per > > provider. Assume no more re-use between plugins than this. > > - Deprecate the current surefire plugin and declare that no new > > parameters EVER will be added to it. All new parameters will only be > > added to the provider-specific plugins. > > - Make the deprecated surefire plugin delegate to the 4 well known new > > mojos (testng, junit3, junit4, junit47), so that the deprecated > > plugin can be kept indefinitely compatible at surefire 2.6 level. Users > > wishing post-2.6 features will have to declare an explicit plugin > > dependency to the specific mojo. > > - Make all surefire mojos implement a specific interface/marker of some > > sort. The old surefire-plugin will have to detect the presence of any > > such mojo; if any such explicit declaration exists, the old surefire > > does nothing. > > - Converting from old-style "surefire" declaration to the new > > per-provider-mojo should basically be about replacing GAV identifier in > > the pom to the provider-specific version, since this "by definition" > > should accept all the same parameters as the existing surefire mojo. > > - The important stuff (extracting the libraries that can be used by > > "anyone") should be doable for 2.7, maybe converting one or two of the > > current providers in the process; I like using the junit47 provider for > > this stuff since it's mostly used by folks running // junit. Porting ALL > > the existing providers to this schema can probably wait until 2.8-2.9 > > ish. An advantage of this approach would be the ability to NOT finalize > > the api's for 2.7 but to be able to adjust this slightly if/when someone > > wants to write an independent provider (or fork one of the existing ones > > - the guy who wanted to do parallel junit 3 comes to mind), and maybe > > aim for a "surefire 3.0" like target for the first officially frozen > > api. > > > > My only uncertainty wrt this second option is how failsafe would fit in; > > maybe Stephen has some thoughts on this... > > > > > > What do you think ? > > > > Kristian > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > > For additional commands, e-mail: dev-h...@maven.apache.org > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
