Peter, Thanks for your feedback. 2009/4/1 Peter Niederwieser <[email protected]>
> > Hi Tom, > > a few quick thoughts: > > - I would expect that I could use multiple test frameworks (e.g. JUnit and > TestNG), and Gradle would still run all of my tests. Will this be > supported? This will be possible but not with autodetection, you'd have to create 2 test tasks and specify which test framework should be used and mention junit in the first and testng in the second. Using one task would cause problems with the different test framework options that available for junit/testng. (This at the moment a problem with the autodetection, I have to look at because the test framework is not known yet at build evaluation, mh tricky -> no testframework detection only test class finding?) > > - Javassist and Mina sound like very heavyweight dependencies "just" to run > tests. Ideally, 20KB of ASM would be enough. I didn't look at ASM yet I'll definately give it a try. Mina is indeed rather large I wanted to use nio for speed but not dig to much in the details but I also have the feeling that it may be overkill. > > - If you are looking for JUnit annotations on a class, the logic should be > "classHasRunWithAnnotation() || anyMethodHasTestAnnotation()". (I'm > mentioning this because most IDEs initially got this wrong and only looked > for @Test.) Thanks for this, I hadn't added this in the detection logic yet. > > - I'm not sure if skipping classes with "$" in their name is completely > safe, even though people probably won't write unit tests as static inner > classes. Is this done for performance reasons? My experience with ASM is > that scanning for annotations is incredibly fast. They only reason I added it was because javassist threw an exception when trying to read those class files. > > - I've done auto-detection of JUnit tests with ASM for Spock. You can have > a > look at it here: > > http://code.google.com/p/spock/source/browse/#svn/trunk/spock-core/src/main/java/org/spockframework/buildsupport > . I'll have a look at it. > > I chose the same ASM version that Groovy uses, even though it's not the > latest one. interesting, keeping the amount of dependencies low is indeed important. > > > Cheers, > Peter > > > Tom Eyckmans wrote: > > > > Hi guys, > > > > I'm working on the native test framework execution stuff, I just want to > > let > > you guys know how I'm trying to implement this so you can contribute > > ideas/ > > better ways of doing things. > > > > Global overview: > > - recurse through compiled test classes, filter out class files that > > contain > > a $ sign => testClassFilesQueue ( BlockingQueue ) > > - testClassFilesQueue => scan for test classes (first found test class > > determines test framework) => testInfosQueue ( BlockingQueue ) > > - testServer / testClient > > - server controls the forked test process: > > - test process requests work (which test to execute next > > (dequeued > > from testInfosQueue -> work/memory throthle), no more work -> terminate) > > - client sends progress events to the server => progressEventQueue > => > > these events get duplicated to BuildListener notifications and Test > Report > > output, which both need to get executed serially. > > > > Smarter test detection. This is done on the compiled test classes. Only > > test > > classes that don't contain a $ sign are queued on a blocking queue for > > processing. > > > > Queued classes are checked by using javassist to get to the information ( > > annotations, parent classes ). The logic required to detect test classes > > is > > located in implementations of the TestFramework interface. I've added > some > > base logic to scan for annotations in the AbstractTestFramework class; > the > > logic in here checks for a number of annotations on methods. As Adam > > mentioned to me the test methods can be inherited and so if no methods > are > > found on the current class it will try and scan upwards on the > inheritance > > tree and stop when the parent class is java.lang.Object. Scanning up the > > inheritance tree is currently only done when the super test class is > > available in the test-classes directory (scanning upwards should be done > > eventually in the complete classpath). > > > > When a class is identified as an actual test class, an implementation of > > TestInfo is queued testInfosQueue, this object is a reference to the test > > class and the type of implementation determines the way the test class > > needs > > to be processed. This is needed to support executing JUnit TestCases and > > TestSuites or TestNG Test classes and xml suite files. > > > > When the first test class is identified the TestFramework that identified > > the test class is used. This causes only that framework to be used for > > test > > detection from that point on. > > > > The only way I could come up with a mechanism to provide the most > > 'real-time' progress notifications and a way to control the forked vm was > > with something networkish => a client-server communication process, > > currently I've implemented this with a java.nio socket client-server with > > Apache MINA. To control the ports used by the server processes (over > > multiple gradle builds running at the same time) I'm currently using a > > single file in ~/.gradle/internal/testing/ports.used that is exclusively > > locked while a new port is being determined or when a port is no longer > > used. Currently I'm starting to use ports from 2000 on, I've just picked > > this port so this may not be the best starting point. Ideally I want to > > detect free ports so there is really no need for configuration for ports. > > > > With this client-server in place the forked vm gets work from the server > > that dequeues work from the testInfosQueue (dequeueing in blocks of > > 100(just > > to name a number) tests). We can also provide a way for the forked vm to > > wait until we want it to start executing tests ( this is usefull when a > > user > > wants to debug the tests, the user specifies -Dtest.debug Gradle asks the > > user to confirm, which may be a first way of solving GRADLE-388, ideally > > I'd > > want to detect when the debugger is attached and proceed then). > > > > Progress events are send by the client and queued on the server side, > this > > can be used to notify BuildListeners and for test output. I'm currently > > undecided where to do the output processing. I'm currently in favour of > > doing this in the gradle build process as to limit the classpath of the > > forked vm. I think we can re-use some of the Ant JUnit output code but > not > > sure if this is something that we want. > > > > I'd love to receive feedback on this. > > > > Thx, > > > > Tom > > > > > > -- > View this message in context: > http://www.nabble.com/Native-test-framework-integration-tp22815266p22820115.html > Sent from the gradle-dev mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > >
