On Fri, May 28, 2010 at 8:57 AM, Mark Hindess <mark.hind...@googlemail.com> wrote: > > In message <aanlktikngv4g0uiqlmhp2d7hcd71l7mxnqoelnizg...@mail.gmail.com>, > Nathan Beyer writes: >> >> On Thu, May 27, 2010 at 9:12 AM, Mark Hindess >> <mark.hind...@googlemail.com> wrote: >> > >> > In message <aanlktikemrmbiyqva2fnpkr4vcvaw7af6lwmig8ns...@mail.gmail.com>, >> > Nathan Beyer writes: >> >> >> >> On Wed, May 26, 2010 at 4:59 AM, Mark Hindess >> >> <mark.hind...@googlemail.com> wrote: >> >> > >> >> > >> >> > Last night I decided to try to implement this simple exclude list >> >> > mechanism in my build-improvement branch. I've checked in my >> >> > changes at >> >> > https://svn.apache.org/repos/asf/harmony/enhanced/java/branches/m...@948377 >> >> >> >> I don't want to be rude, >> > >> > Not at all; I don't think you've ever been rude on this list. >> > >> >> but ... I brought this up a few months back and the response I >> >> got was less than overwhelming. I stopped my work. How is this >> >> different? >> >> >> >> http://harmony.markmail.org/message/4rhomwqjdr5uklln?q=3Djunit+annotation >> > >> > I think the response you got was fairly positive. The only issue >> > appears to be a request from Jesse for some way to externally >> > influence the test selection process which I don't think was >> > tackled. >> > >> > How far did you get? I'd be interested in playing with the >> > annotations if you've written them. I'm not convinced annotations >> > are enough but they probably will be part of the right longer term >> > solution. >> >> What I have now is - >> * Platform - an annotation that can mark a class and/or method with a >> os, arch and vm value >> * PlatformRule - a org.junit.rules.MethodRule implementation, which >> is instantiated and stored as an instance variable in unit tests that >> use the Platform annotation; when a test with this rule is run, if the >> class or method is annotated with Platform, only the test methods that >> match all qualifications will be run; methods not run will be logged >> to standard error. > > It would be nice to be able to do things like: > > �...@exclude() // for exclude.{common,intern} tests > �...@exclude(os=linux) // for tests that fail on any linux platform > > but to start with so long as we can have multiple @Exclude annotation > this wont be an urgent problem to solve. > >> This was my experiment for using annotations and method rules. The >> next step would be to define some annotations for the test >> qualifications we wanted to express - Exclude(os, arch, vm), etc. And >> then write method rules that would massage the run based on these >> rules. > > I assume the Exclude annotation wont be a big leap from the Platform > annotation. If you can provide this I can use a similar Perl one-liner > to the one I used for the Support_Excludes changes in my branch to apply > them to all methods in classes in the current exclude lists. We can > then refine them after checking which methods really need to be excluded > as we get time to go through them. > >> The interesting thing about method rules is that they can affect >> the test runs without requiring any special runner, however, when a >> method rule is used to not run a method, most runners will treat the >> methods as run, even if the method rule doesn't actually execute the >> 'statement'. > > Method rules would be good enough for now but having the top-level > junit report showing more tests run even though they might have tested > nothing is not ideal. So, I think longer term a runner will be > necessary/useful. (I'd expect Jesse would override the runner with his > own to get whatever external filtering he needs.)
Here's the issue -- the JUnit Ant Task doesn't provide much access to the flexibility in JUnit4's APIs. Anything that gets into the realm of a custom runner or customizing the run (filtering, etc) would require a custom Ant task. If we're beholden to using the standard task, then we're limited in what can be done. I think defining custom, semantic annotations for the tests is the best approach for Harmony (both for excludes, but also for includes -- i.e. run test for specific platform), but to get them working we'll have to write, at a minimum, a custom Ant task for running the tests. -Nathan > > Regards, > Mark. > > >
