I just thought about the package naming convention idea a bit further. Isn't such an environment constraint always something like "requires at least X"?
In this case we could do it this way: If a test requires at least EE-light, it is placed in a package *.eelight.* within the testsuite. All containers profiles that don't support the minimum requirement (e.g. weld-embedded or tomcat) exclude this package in the Surefire configuration. This would be very simply to set up. It's a bit similar to what Seam3 currently does but the idea is based on environments instead of containers and the "requires at least" concept makes it very simple to use. Thoughts? Christian 2011/12/17 Jason Porter <[email protected]>: > I like the idea, however, that would be a very core change to both core > arquillian and the containers. I don't think we'll get that anytime soon. > Maybe a simple package convention by environment will work for now? > > Sent from my iPhone > > On Dec 17, 2011, at 9:19, Mark Struberg <[email protected]> wrote: > >> true, we would need something like >> >> @RunInEnvironments({Environment.EeLight, Environment.EeFull, >> Environment.Servlet}) >> >> @RunInEnvironments(Environment.JavaSE) >> @RunInEnvironments(Environments.ALL) >> >> LieGrue, >> strub >> >> >> >> ----- Original Message ----- >>> From: Christian Kaltepoth <[email protected]> >>> To: [email protected] >>> Cc: >>> Sent: Saturday, December 17, 2011 5:04 PM >>> Subject: Re: basic decisions: test setup >>> >>> @Mark: >>> >>> But what about tests that should run in EE-light, EE-full and Servlet >>> but not in SE. Or tests that should run in EE-light/full but not in >>> Servlet and SE. I think this could get very complex. >>> >>> Christian >>> >>> >>> 2011/12/17 Mark Struberg <[email protected]>: >>>> We might solve this simply with a package naming rule. And then we >>> configure surefire to just run the tests with the respective package? >>>> >>>> Not the most advanced solution, but will work. >>>> >>>> >>>> LieGrue, >>>> strub >>>> >>>> >>>> >>>> ----- Original Message ----- >>>>> From: Christian Kaltepoth <[email protected]> >>>>> To: [email protected] >>>>> Cc: >>>>> Sent: Saturday, December 17, 2011 4:43 PM >>>>> Subject: Re: basic decisions: test setup >>>>> >>>>> @Mark: >>>>> >>>>> Actually @TargetsContainer is an existing feature of Arquillian. It >>>>> allows to build container specific deployments. >>>>> >>>>> https://docs.jboss.org/author/display/ARQ/Multiple+Containers >>>>> >>>>> Unfortunately it won't fit our needs because the test completely >>> fails >>>>> if there is no matching deployment for the container the test is >>>>> executed against. So I guess we have to build something new for this. >>>>> >>>>> I really like your suggestion regarding the four environments. It >>>>> absolutely make sense to declare environment constraints (like SE, >>>>> Servlet, EE, etc.) instead of listing concrete containers. >>>>> >>>>> Christian >>>>> >>>>> >>>>> >>>>> 2011/12/17 Mark Struberg <[email protected]>: >>>>>> Hi Christian! >>>>>> >>>>>> >>>>>> @TargetContainer sounds good, but I'd rather make it a >>>>>> >>>>>> >>>>>> @TargetEnvironment() >>>>>> >>>>>> with values: SE, Servlet, EE-light, EE-full >>>>>> >>>>>> The excludes should be done in the single test integration >>> modules, and the >>>>> goal should be to get down to zero excluded tests for all containers. >>>>>> >>>>>> >>>>>> @Jason: that mixture for the tests looks fine to me as well. Can >>> you please >>>>> hack a sample structure in the DeltaSpikePlayground git repo? >>>>>> >>>>>> >>>>>> LieGrue, >>>>>> strub >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> ________________________________ >>>>>>> From: Christian Kaltepoth <[email protected]> >>>>>>> To: [email protected] >>>>>>> Sent: Saturday, December 17, 2011 7:55 AM >>>>>>> Subject: Re: basic decisions: test setup >>>>>>> >>>>>>> I would like to add some "non-binding" comments here >>> regarding >>>>> 3). >>>>>>> >>>>>>> Actually I really like the concept Jason described and which is >>>>>>> currently used in Seam3. Having worked with it a bit I can >>> confirm >>>>>>> that it runs fine. And somehow it's the best of both worlds >>> (in >>>>> regard >>>>>>> to 3a vs 3b). >>>>>>> >>>>>>> That the full integration test suite with all containers cannot >>> run in >>>>>>> a single Maven invocation is IMHO not a real problem. We could >>> write a >>>>>>> simple script for this. >>>>>>> >>>>>>> The only thing that could be optimized in this setup is the >>> separation >>>>>>> of container dependent tests via package name. But perhaps this >>> could >>>>>>> be solved by Arquillian? What I would really like to see is the >>>>>>> ability to configure a test so it is only executed in a >>> specific >>>>>>> container (something similar to @TargetsContainer). This way >>> tests >>>>>>> that require to be run in a full container could be executed >>> only in >>>>>>> JBoss and Glassfish but would be skipped in embedded Weld. >>>>>>> >>>>>>> Christian >>>>>>> >>>>>>> >>>>>>> 2011/12/16 Jason Porter <[email protected]>: >>>>>>>> Comments inline. >>>>>>>> >>>>>>>> On Fri, Dec 16, 2011 at 01:38, Mark Struberg >>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi! >>>>>>>>> >>>>>>>>> Another round of discussion - this time about how to >>> setup our >>>>> unit >>>>>>>>> testing and integration testing. >>>>>>>>> >>>>>>>>> I think it's pretty much clear that we will gonna >>> use >>>>> Arquillian for all >>>>>>>>> our tests. That was also fundamental part of the >>> incubator >>>>> proposal. >>>>>>>>> And instead of inventing yet another abstraction >>> layer, I'd >>>>> favour to just >>>>>>>>> use Arquillian and contribute all things we need to >>> this >>>>> project. >>>>>>>>> So here a formal vote >>>>>>>>> >>>>>>>>> 1.) >>>>>>>>> >>>>>>>>> for using Arquillian as test integration framework. >>>>>>>>> +1 from me >>>>>>>>> >>>>>>>> >>>>>>>> +1 from me >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> 2.) >>>>>>>>> >>>>>>>>> What do we like to use from the unit tests itself? >>> TestNG or >>>>> JUnit? >>>>>>>>> JUnit has better Arquillian integration and TestNG is >>> better >>>>> for 'real >>>>>>>>> world' projects, because it allows to define test >>>>> dependencies. >>>>>>>>> So which one to take? >>>>>>>>> But we should definitely only use 1 of the two >>> exclusively! >>>>>>>>> >>>>>>>> >>>>>>>> +1 for using one exclusively >>>>>>>> >>>>>>>> As for which, I'm also +0 I like both for different >>> reasons, >>>>> but JUnit is >>>>>>>> certainly the easier of the two to use with Arquillian. >>>>>>>> >>>>>>>> Also, are we open to having other languages for tests or >>> using >>>>> other >>>>>>>> testing structures such as spock (it works with >>> Arquillian, BTW), >>>>> RSpec, >>>>>>>> scala, etc.? >>>>>>>> >>>>>>>> >>>>>>>>> 3.) >>>>>>>>> 'Integrated' tests vs 'Integration >>> Tests' >>>>>>>>> We have 2 options to test our projects >>>>>>>>> >>>>>>>>> 3.a.) create a full unit test in each module (e.g. >>>>> deltaspike/core/impl) >>>>>>>>> and add a profile for each and every server (maybe we >>> can trim >>>>> this down >>>>>>>>> with pom imports?). >>>>>>>>> Then run the build with one after each other: >>>>>>>>> >>>>>>>>> $> mvn clean test -Powb >>>>>>>>> $> mvn clean test -Pweld >>>>>>>>> $> mvn clean test -Powb-tc >>>>>>>>> $> mvn clean test -Pweld-tc >>>>>>>>> $> mvn clean test -Pgf31 >>>>>>>>> $> mvn clean test -Pas7 >>>>>>>>> $> mvn clean test -Ptomee >>>>>>>>> >>>>>>>>> This might pretty much blow up our poms... >>>>>>>>> >>>>>>>>> >>>>>>>>> 3.b) create a full unit test in each module (e.g. >>>>> deltaspike/core/impl) >>>>>>>>> and add only 2 profiles directly ('weld' and >>>>> 'owb' (probably resin later)) >>>>>>>>> Then add a deltaspike/test/base/core with the very >>> basic parent >>>>> stuff and >>>>>>>>> 1 module for each integrated container, e.g. >>>>>>>>> deltaspike/test/weld-tc >>>>>>>>> deltaspike/test/owb-tc >>>>>>>>> deltaspike/test/weld-jetty >>>>>>>>> deltaspike/test/owb-jetty >>>>>>>>> deltaspike/test/tomee >>>>>>>>> deltaspike/test/gf31 >>>>>>>>> deltaspike/test/geronimo >>>>>>>>> deltaspike/test/websphere >>>>>>>>> deltaspike/test/as7 >>>>>>>>> deltaspike/test/as6 >>>>>>>>> etc. >>>>>>>>> >>>>>>>>> The initial setup costs are higher, but it would be >>> pretty easy >>>>> to add new >>>>>>>>> containers that way. >>>>>>>>> >>>>>>>>> Which route shall we take? >>>>>>>>> >>>>>>>> >>>>>>>> In Seam 3 we have an approach which is a little bit of a >>> mixture of >>>>> both. >>>>>>>> We have regular unit tests (with mockito, stubs, etc) >>> under >>>>>>>> impl/src/test/... and then we have a separate module >>> called >>>>> testsuite which >>>>>>>> contained all the integration tests. We used pom imports >>> to get all >>>>> the >>>>>>>> containers setup correctly and then had different profiles >>> which >>>>> pulled in >>>>>>>> the poms and also configured surefire for that server. >>> Inside the >>>>> code >>>>>>>> structure we separated the servers by package: common, >>> jbossas, >>>>> glassfish, >>>>>>>> etc. That way all of the tests were in one location, they >>> were easy >>>>> to >>>>>>>> execute, the tests stayed DRY, and the IDE wasn't much >>> of a >>>>> problem. The >>>>>>>> two downsides we had were 1) the extra config in the pom >>> and 2) you >>>>> can't >>>>>>>> run the full integration suite in one maven execution. We >>> spent >>>>> quite a bit >>>>>>>> of time examining options and trying some of them out. >>> This was the >>>>> best we >>>>>>>> came up with, and also the one Aslak thought would work >>> best as >>>>> well. >>>>>>>> >>>>>>>> >>>>>>>>> LieGrue, >>>>>>>>> strub >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Jason Porter >>>>>>>> http://lightguard-jp.blogspot.com >>>>>>>> http://twitter.com/lightguardjp >>>>>>>> >>>>>>>> Software Engineer >>>>>>>> Open Source Advocate >>>>>>>> Author of Seam Catch - Next Generation Java Exception >>> Handling >>>>>>>> >>>>>>>> PGP key id: 926CCFF5 >>>>>>>> PGP key available at: keyserver.net, pgp.mit.edu >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Christian Kaltepoth >>>>>>> Blog: http://chkal.blogspot.com/ >>>>>>> Twitter: http://twitter.com/chkal >>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Christian Kaltepoth >>>>> Blog: http://chkal.blogspot.com/ >>>>> Twitter: http://twitter.com/chkal >>>>> >>> >>> >>> >>> -- >>> Christian Kaltepoth >>> Blog: http://chkal.blogspot.com/ >>> Twitter: http://twitter.com/chkal >>> -- Christian Kaltepoth Blog: http://chkal.blogspot.com/ Twitter: http://twitter.com/chkal
