On Feb 15, 2012, at 4:10 PM, Vincent Massol wrote: > > On Jan 25, 2012, at 5:02 PM, Vincent Massol wrote: > >> >> On Jan 15, 2012, at 11:27 AM, Vincent Massol wrote: >> >>> Hi devs, >>> >>> I've started an experiment to have colocated functional tests (CFT), which >>> means having the functional tests located where the functional domain >>> sources are located instead of in XE. >>> >>> For example for the linkchecker module we have the following directories: >>> >>> xwiki-platform-linkchecker/ >>> |_ xwiki-platform-linkchecker-refresher (JAR) >>> |_ xwiki-platform-linkchecker-ui (XAR) >>> |_ xwiki-platform-linkchecker-tests (functional tests) >>> >>> The rationale for this was: >>> * Have everything about a functional domain self-contained (source and all >>> tests) >>> * Making it easy to run only tests for a given functional domain >>> * Move page objects to the functional domain too >>> >>> Here are some findings about this experiment: >>> >>> A - It takes about 30 seconds to generate the adhoc packaging and start >>> XWiki. This would be done for each module having functional tests compared >>> to only once if all tests were executed in XE >>> B- The package mojo created to generate a full packaging is quite nice and >>> I plan to reuse it in lots of other places in our build (distributions, >>> database, places where we need XWiki configuration files) >>> C- We will not be able to run platform builds in Maven multithreaded mode >>> since it would mean that several XWiki instance could be started at the >>> same time on the same port >>> D- The colocated functional test module >>> >>> Solutions/ideas: >>> >>> * One idea to overcome A and C would to have the following setup: >>> ** Keep functional test modules colocated but have them generate a test JAR >>> ** Still allow running functional tests from the colocated module (this >>> makes it easy to verify no regression was introduced when making changes to >>> a given domain) >>> ** Have functional tests in XE depend on the colocated functional test >>> module JARs and configure Jenkins to run all functional tests from XE only >>> >>> * Another solution to overcome C is to auto-discover the port to use in our >>> XWiki startup script (and save it in a file so that the stop script can use >>> it). >>> >>> I think the first proposal is the best one and brings the best of the 2 >>> worlds. >> >> Actually there's another one: >> >> * Run functional tests in platform >> * Configure jenkins so that we have one jenkins job per platform top module >> >> This would allow several things: >> * Faster feedback to user: for example if a commit is done in a module that >> happens late in the build order the whole platform wouldn't need to be >> rebuilt before noticing the problem) - Note: Jenkins supports a "Local >> subdirectory for repo" feature that would make this possible I think) > > After researching this a bit I don't think it's possible… > > This "Local subdirectory for repo" is something else: "Specify a local > directory (relative to the workspace root) where the Git repository will be > checked out. If left empty, the workspace root itself will be used." > > We could check out the whole Git repo but 1) it'll take space (a lot of > space) but more importantly 2) GitHub will trigger a build on all jobs for > that repo which isn't good. > > So I think we'll have to drop the idea of having a single job per module. > > See also > http://stackoverflow.com/questions/8922857/jenkins-how-to-build-multiple-top-level-projects-from-a-git-repository
hmm actually this is what we already do for functional tests but we don't trigger than on Git Hub notifications, we trigger them on xwiki-enterprise (after it's been built). I guess we could do the same, i.e.: * Only trigger the top level module in a given repo using a GitHub push * Only trigger the other modules on "Build whenever a SNAPSHOT dependency is built" The problem is that it would still use a lot of space. For example my xwiki-platform/ dir takes 1.2GB of disk space (with target/ dirs) So with roughly 230+ modules that's 276 GB just for xwiki-platform… Combined with commons, rendering and enterprise that's probably around 500GB easily… In conclusion it doesn't seem like a good idea… Thanks -Vincent > Thanks > -Vincent > >> * Use the full power of our agents by distributing the jobs >> * Since functional tests run at the same time as unit tests if the module >> build works we can be sure it's good >> >> Thus even though packaging and starting XE takes a lot of time (I've timed >> it to roughly 1.5 minutes on my machine) the balancing of jobs on multiple >> agents might overcome this - it would still be great to be able to improve >> the XE start time. >> >> Note that creating jenkins job for so many modules might seem daunting but >> this can be easily automated with http://ci.xwiki.org/api/ or even better >> http://evgeny-goldin.com/wiki/Maven-jenkins-plugin >> >> I'm still hesitating between the previous idea and this one but thought I >> would share my current thoughts with you in case you have some >> idea/preference… :) >> >> Thanks >> -Vincent >> >>> WDYT? >>> >>> Thanks >>> -Vincent >>> >> > _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
