I like the annotation filter! (Don't yet understand why isn't that a Qualifier but a Stereotype, but that's minor stuff)
LieGrue, strub ----- Original Message ----- > From: Dan Allen <[email protected]> > To: [email protected] > Cc: > Sent: Friday, December 23, 2011 2:10 AM > Subject: Re: basic decisions: test setup > > You've hit a topic very near and dear to my heart, runtime environment > compatibility filters. > > Awhile ago I laid out a proposal for this feature *very* similar to what > Mark suggested (e.g., @RunInEnvironments). You can review the proposal here: > > http://community.jboss.org/thread/156500 > > After many discussions with Aslak, what we realized that the both > environment qualifiers (e.g., JavaEE6Web) and the binding to a container > implementation (e.g., JBoss AS 7) really need to be defined by the user (as > opposed to defined in Arquillian). What Arquillian needs to provide is the > mechanism for filtering the test cases that aren't suitable for the current > container being used. > > Our latest idea, concocted during Devoxx, is as follows: > > You define a qualifier annotation that represents a type of environment > (e.g., typically some sort of specification) > > @Retention(RetentionPolicy.RUNTIME) > @Target(ElementType.TYPE) > @ComplianceStereotype > public @interface JavaEE6Web {} > > (I suppose the annotation could instead be an enum as well...not sure the > right choice). > > Then you annotate your test class to indicate the requirement. > > @RunWith(Arquillian.class) > @RunsInEnvironment(JavaEE6Web.class) > public class JavaEE6WebSpecificTestCase { > ... > } > > Now, so far, nothing special would happen. Arquillian would have no way to > determine whether the current container is compatible with the JavaEE6Web > stereotype. For that, we have to tell Arquillian about the relationship via > arquillian.xml: > > <arquillian> > <container qualifier="jbossas-remote-7"> > <compliance > qualifier="org.example.compliance.JavaEE6Web"/> > ... > </container> > </arquillian> > > > Naturally, since Arquillian can be configured using Java, you could > establish these bindings in a Java class, and thus via a JAR if you wanted. > > The point is, Arquillian would know whether a certain container is > compliant with the stereotype specified on the test class. If it isn't, the > test is skipped (or otherwise filtered out). > > I would definitely push to get this feature into Arquillian because I've > long believed it's a very important feature. While you can implement your > own test filtering based on package names (or TestNG groups), that's > exactly the type of work that Arquillian tries to eliminate from your > plate. If a test couldn't possibly work on a given container, then > Arquillian shouldn't even attempt to run it. > > -Dan > > p.s. We could also support groups/inheritence, so that JavaEE 6 Web > compliance would infer CDI compliance. In that case, this container would > also match the org.example.compliance.CDI environment stereotype. > > >> I like the idea, however, that would be a very core change to both core >> arquillian and the containers. I don't think we'll get that anytime > soon. >> Maybe a simple package convention by environment will work for now? >> >> Sent from my iPhone >> >> On Dec 17, 2011, at 9:19, Mark Struberg <[email protected]> wrote: >> >> > true, we would need something like >> > >> > @RunInEnvironments({Environment.EeLight, Environment.EeFull, >> Environment.Servlet}) >> > >> > @RunInEnvironments(Environment.JavaSE) >> > @RunInEnvironments(Environments.ALL) >> > >> > LieGrue, >> > strub >> > >> > >> > >> > ----- Original Message ----- >> >> From: Christian Kaltepoth <[email protected]> >> >> To: [email protected] >> >> Cc: >> >> Sent: Saturday, December 17, 2011 5:04 PM >> >> Subject: Re: basic decisions: test setup >> >> >> >> @Mark: >> >> >> >> But what about tests that should run in EE-light, EE-full and > Servlet >> >> but not in SE. Or tests that should run in EE-light/full but not > in >> >> Servlet and SE. I think this could get very complex. >> >> >> >> Christian >> >> >> >> >> >> 2011/12/17 Mark Struberg <[email protected]>: >> >>> We might solve this simply with a package naming rule. And > then we >> >> configure surefire to just run the tests with the respective > package? >> >>> >> >>> Not the most advanced solution, but will work. >> >>> >> >>> >> >>> LieGrue, >> >>> strub >> >>> >> >>> >> >>> >> >>> ----- Original Message ----- >> >>>> From: Christian Kaltepoth <[email protected]> >> >>>> To: [email protected] >> >>>> Cc: >> >>>> Sent: Saturday, December 17, 2011 4:43 PM >> >>>> Subject: Re: basic decisions: test setup >> >>>> >> >>>> @Mark: >> >>>> >> >>>> Actually @TargetsContainer is an existing feature of > Arquillian. It >> >>>> allows to build container specific deployments. >> >>>> >> >>>> > https://docs.jboss.org/author/display/ARQ/Multiple+Containers >> >>>> >> >>>> Unfortunately it won't fit our needs because the test > completely >> >> fails >> >>>> if there is no matching deployment for the container the > test is >> >>>> executed against. So I guess we have to build something > new for this. >> >>>> >> >>>> I really like your suggestion regarding the four > environments. It >> >>>> absolutely make sense to declare environment constraints > (like SE, >> >>>> Servlet, EE, etc.) instead of listing concrete containers. >> >>>> >> >>>> Christian >> >>>> >> >>>> >> >>>> >> >>>> 2011/12/17 Mark Struberg <[email protected]>: >> >>>>> Hi Christian! >> >>>>> >> >>>>> >> >>>>> @TargetContainer sounds good, but I'd rather make > it a >> >>>>> >> >>>>> >> >>>>> @TargetEnvironment() >> >>>>> >> >>>>> with values: SE, Servlet, EE-light, EE-full >> >>>>> >> >>>>> The excludes should be done in the single test > integration >> >> modules, and the >> >>>> goal should be to get down to zero excluded tests for all > containers. >> >>>>> >> >>>>> >> >>>>> @Jason: that mixture for the tests looks fine to me > as well. Can >> >> you please >> >>>> hack a sample structure in the DeltaSpikePlayground git > repo? >> >>>>> >> >>>>> >> >>>>> LieGrue, >> >>>>> strub >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>>> ________________________________ >> >>>>>> From: Christian Kaltepoth > <[email protected]> >> >>>>>> To: [email protected] >> >>>>>> Sent: Saturday, December 17, 2011 7:55 AM >> >>>>>> Subject: Re: basic decisions: test setup >> >>>>>> >> >>>>>> I would like to add some "non-binding" > comments here >> >> regarding >> >>>> 3). >> >>>>>> >> >>>>>> Actually I really like the concept Jason described > and which is >> >>>>>> currently used in Seam3. Having worked with it a > bit I can >> >> confirm >> >>>>>> that it runs fine. And somehow it's the best > of both worlds >> >> (in >> >>>> regard >> >>>>>> to 3a vs 3b). >> >>>>>> >> >>>>>> That the full integration test suite with all > containers cannot >> >> run in >> >>>>>> a single Maven invocation is IMHO not a real > problem. We could >> >> write a >> >>>>>> simple script for this. >> >>>>>> >> >>>>>> The only thing that could be optimized in this > setup is the >> >> separation >> >>>>>> of container dependent tests via package name. But > perhaps this >> >> could >> >>>>>> be solved by Arquillian? What I would really like > to see is the >> >>>>>> ability to configure a test so it is only executed > in a >> >> specific >> >>>>>> container (something similar to > @TargetsContainer). This way >> >> tests >> >>>>>> that require to be run in a full container could > be executed >> >> only in >> >>>>>> JBoss and Glassfish but would be skipped in > embedded Weld. >> >>>>>> >> >>>>>> Christian >> >>>>>> >> >>>>>> >> >>>>>> 2011/12/16 Jason Porter > <[email protected]>: >> >>>>>>> Comments inline. >> >>>>>>> >> >>>>>>> On Fri, Dec 16, 2011 at 01:38, Mark Struberg >> >>>> <[email protected]> wrote: >> >>>>>>> >> >>>>>>>> Hi! >> >>>>>>>> >> >>>>>>>> Another round of discussion - this time > about how to >> >> setup our >> >>>> unit >> >>>>>>>> testing and integration testing. >> >>>>>>>> >> >>>>>>>> I think it's pretty much clear that > we will gonna >> >> use >> >>>> Arquillian for all >> >>>>>>>> our tests. That was also fundamental part > of the >> >> incubator >> >>>> proposal. >> >>>>>>>> And instead of inventing yet another > abstraction >> >> layer, I'd >> >>>> favour to just >> >>>>>>>> use Arquillian and contribute all things > we need to >> >> this >> >>>> project. >> >>>>>>>> So here a formal vote >> >>>>>>>> >> >>>>>>>> 1.) >> >>>>>>>> >> >>>>>>>> for using Arquillian as test integration > framework. >> >>>>>>>> +1 from me >> >>>>>>>> >> >>>>>>> >> >>>>>>> +1 from me >> >>>>>>> >> >>>>>>> >> >>>>>>>> >> >>>>>>>> 2.) >> >>>>>>>> >> >>>>>>>> What do we like to use from the unit > tests itself? >> >> TestNG or >> >>>> JUnit? >> >>>>>>>> JUnit has better Arquillian integration > and TestNG is >> >> better >> >>>> for 'real >> >>>>>>>> world' projects, because it allows to > define test >> >>>> dependencies. >> >>>>>>>> So which one to take? >> >>>>>>>> But we should definitely only use 1 of > the two >> >> exclusively! >> >>>>>>>> >> >>>>>>> >> >>>>>>> +1 for using one exclusively >> >>>>>>> >> >>>>>>> As for which, I'm also +0 I like both for > different >> >> reasons, >> >>>> but JUnit is >> >>>>>>> certainly the easier of the two to use with > Arquillian. >> >>>>>>> >> >>>>>>> Also, are we open to having other languages > for tests or >> >> using >> >>>> other >> >>>>>>> testing structures such as spock (it works > with >> >> Arquillian, BTW), >> >>>> RSpec, >> >>>>>>> scala, etc.? >> >>>>>>> >> >>>>>>> >> >>>>>>>> 3.) >> >>>>>>>> 'Integrated' tests vs > 'Integration >> >> Tests' >> >>>>>>>> We have 2 options to test our projects >> >>>>>>>> >> >>>>>>>> 3.a.) create a full unit test in each > module (e.g. >> >>>> deltaspike/core/impl) >> >>>>>>>> and add a profile for each and every > server (maybe we >> >> can trim >> >>>> this down >> >>>>>>>> with pom imports?). >> >>>>>>>> Then run the build with one after each > other: >> >>>>>>>> >> >>>>>>>> $> mvn clean test -Powb >> >>>>>>>> $> mvn clean test -Pweld >> >>>>>>>> $> mvn clean test -Powb-tc >> >>>>>>>> $> mvn clean test -Pweld-tc >> >>>>>>>> $> mvn clean test -Pgf31 >> >>>>>>>> $> mvn clean test -Pas7 >> >>>>>>>> $> mvn clean test -Ptomee >> >>>>>>>> >> >>>>>>>> This might pretty much blow up our > poms... >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> 3.b) create a full unit test in each > module (e.g. >> >>>> deltaspike/core/impl) >> >>>>>>>> and add only 2 profiles directly > ('weld' and >> >>>> 'owb' (probably resin later)) >> >>>>>>>> Then add a deltaspike/test/base/core with > the very >> >> basic parent >> >>>> stuff and >> >>>>>>>> 1 module for each integrated container, > e.g. >> >>>>>>>> deltaspike/test/weld-tc >> >>>>>>>> deltaspike/test/owb-tc >> >>>>>>>> deltaspike/test/weld-jetty >> >>>>>>>> deltaspike/test/owb-jetty >> >>>>>>>> deltaspike/test/tomee >> >>>>>>>> deltaspike/test/gf31 >> >>>>>>>> deltaspike/test/geronimo >> >>>>>>>> deltaspike/test/websphere >> >>>>>>>> deltaspike/test/as7 >> >>>>>>>> deltaspike/test/as6 >> >>>>>>>> etc. >> >>>>>>>> >> >>>>>>>> The initial setup costs are higher, but > it would be >> >> pretty easy >> >>>> to add new >> >>>>>>>> containers that way. >> >>>>>>>> >> >>>>>>>> Which route shall we take? >> >>>>>>>> >> >>>>>>> >> >>>>>>> In Seam 3 we have an approach which is a > little bit of a >> >> mixture of >> >>>> both. >> >>>>>>> We have regular unit tests (with mockito, > stubs, etc) >> >> under >> >>>>>>> impl/src/test/... and then we have a separate > module >> >> called >> >>>> testsuite which >> >>>>>>> contained all the integration tests. We used > pom imports >> >> to get all >> >>>> the >> >>>>>>> containers setup correctly and then had > different profiles >> >> which >> >>>> pulled in >> >>>>>>> the poms and also configured surefire for > that server. >> >> Inside the >> >>>> code >> >>>>>>> structure we separated the servers by > package: common, >> >> jbossas, >> >>>> glassfish, >> >>>>>>> etc. That way all of the tests were in one > location, they >> >> were easy >> >>>> to >> >>>>>>> execute, the tests stayed DRY, and the IDE > wasn't much >> >> of a >> >>>> problem. The >> >>>>>>> two downsides we had were 1) the extra config > in the pom >> >> and 2) you >> >>>> can't >> >>>>>>> run the full integration suite in one maven > execution. We >> >> spent >> >>>> quite a bit >> >>>>>>> of time examining options and trying some of > them out. >> >> This was the >> >>>> best we >> >>>>>>> came up with, and also the one Aslak thought > would work >> >> best as >> >>>> well. >> >>>>>>> >> >>>>>>> >> >>>>>>>> LieGrue, >> >>>>>>>> strub >> >>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> Jason Porter >> >>>>>>> http://lightguard-jp.blogspot.com >> >>>>>>> http://twitter.com/lightguardjp >> >>>>>>> >> >>>>>>> Software Engineer >> >>>>>>> Open Source Advocate >> >>>>>>> Author of Seam Catch - Next Generation Java > Exception >> >> Handling >> >>>>>>> >> >>>>>>> PGP key id: 926CCFF5 >> >>>>>>> PGP key available at: keyserver.net, > pgp.mit.edu >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Christian Kaltepoth >> >>>>>> Blog: http://chkal.blogspot.com/ >> >>>>>> Twitter: http://twitter.com/chkal >> >>>>>> >> >>>>>> >> >>>>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Christian Kaltepoth >> >>>> Blog: http://chkal.blogspot.com/ >> >>>> Twitter: http://twitter.com/chkal >> >>>> >> >> >> >> >> >> >> >> -- >> >> Christian Kaltepoth >> >> Blog: http://chkal.blogspot.com/ >> >> Twitter: http://twitter.com/chkal >> >> >> >> >> > > > -- > Dan Allen > Principal Software Engineer, Red Hat | Author of Seam in Action > Registered Linux User #231597 > > http://www.google.com/profiles/dan.j.allen#about > http://mojavelinux.com > http://mojavelinux.com/seaminaction >
