Thx a lot for your swift reply. Hans:> "..expensive operations are done at configuration time which would be better delayed to execution time"
I quickly found one by chance. We have a custom test task that replaces the default java plugin test task. This task is of type JavaExec. By moving setting of systemproperties from config to a doFirst clause I'm now down to approx 4 seconds. ... one of the systemproperties I set is using configurations.testRuntime.asPath, I have a feeling this was the culprit causing all the dependency resolution reports in the log as well :-) I feel quite a bit more confident now on proceeding with adding the remaining subprojects to the multiproject build ! cheers Magnus - A very happy gradle user so far Date: Mon, 6 Sep 2010 15:07:32 +0200 From: [email protected] To: [email protected] Subject: Re: [gradle-user] gradle performance/scalability with lots of subprojects Hi Magnus, On Mon, Sep 6, 2010 at 2:28 PM, Magnus Rundberget <[email protected]> wrote: Hi, We've chosen gradle for our new build system and we've started the journey from maven 1 (!) to gradle. Excellent. We have a very granular structure at the moment, with about 80-90 leaf projects producing output artifacts (and obviously lots of interdependencies going on + lots of 3rd party dependencies). In the short term I don't think restructuring our project structure and associated directory structure is an option. In the longer term it will be consolidated. I've ported about 40 of the simpler projects to one large gradle multiproject build. However there is starting to be a fairly substantial overhead in "initialization" before actually doing anything in terms of building. Doing gradle -t on a subprojsct now takes around 10,5 seconds, same for gradle clean. Usually a clean is faster. -t is a pretty expensive operation as it needs to builds the full DAG. I've tried the same on the gradle branch (Im obviously aware its work in progress) using a gradle daemon. gradle -t on a subproject here takes about 6,5 seconds (after running it 2 times first:- As said above. Rather use clean as a measure. Little more background; - I'm currently using a locally installed nexus repository for all my third party dependencies - I'm running cygwin on windows xp - Using gradle 0.9 rc1 >From running gradle -t -i, I've made the following observations from the log; - Evaluate root project uses about 2 seconds - Summing up resolution report resolve is almost 4 seconds (typically something like "resolution report :: resolve 266ms :: artifacts dl 0ms") The questions; 1) Should I assume that the overhead will get exponentially worse if adding the remaining 40-50 subprojects ? As Robert pointed out, not exponentially but there is an overhead per project. Let's have a closer look at this. Right now Gradle executes all the configuration (e.g. the configuration closures of task) independent of what gets executed. This means even if you do a partial build of a subproject or you just do a clean, every build script of all subprojects will be fully evaluated. There are a couple of things we have in the pipeline to improve this. 1.) The upcoming new configuration mechanism: http://docs.codehaus.org/display/GRADLE/Gradle+1.0+Build+Configuration+DSL With this DSL a lot of the configuration will be lazy and only executed if it is really needed. 2.) In the future Gradle will cache the configured build as much as possible. This will obviously dramatically reduce the configuration time.This is not completely trivial due to the dynamic degrees of freedom you have. For example a naive caching can't deal properly with stuff like: version = 1.0 + '-' + System.currentTimeMillis() With the dependency DSL from 1.) Gradle can provide a rich declarative language from which Gradle can derive what can be cached and what not. 3.) Further profiling to speed up the mechanics of the configuration/execution. 4.) The daemon will be a dramatic improvement as well. 2) Anything I could do to reduce the overhead whilst still getting the benefits of incremental builds (ie without having to resort to splitting into many more multiprojects?) A thing I observe frequently is that expensive operations are done at configuration time which would be better delayed to execution time and be part of the execution of a specific task. There are massive enterprise builds using Gradle with many hundreds subprojects. 3) When can I hope for a working version of 0.9.1 with gradle daemon working on windows ? :-) 0.9.1 will be a dedicated release just for the daemon feature. This means it will be released short after 0.9. - Hans --Hans Dockter Founder, Gradlehttp://www.gradle.org, http://twitter.com/gradleorg CEO, Gradle Inc. - Gradle Training, Support, Consultinghttp://www.gradle.biz
