Cool stuff !
Just tested 1.0-milestone-6-20111106000016+0100Unfortunately I had trimmed down
the number of repositories and snapshot usage (after milestone4 and partially
still milestone 5 resolution performance).However I could still see a
improvement (albeit small compared to groovy compilation times) in dependency
resolution performance.
I must have missed something in the versioning strategy stuff or a bug has been
introduced.
Suspicious behavior:- In my build script I have set up dependency configuration
for groovy(-all) to 1.8.3- I have a testCompile dependency to spock which again
has a transitive dependency to a snapshot version (1.8.0-SNAPSHOT) of
groovy-all- When running my build with milestone-6 it complains that it can't
find the 1.8.0 SNAPSHOT version- Disabling the transitive deps for the spock
libs ensures it works.
So I'm guessing there might be a slight regression going on here (latest
version strategy not handling transitives with snapshot versions perhaps) ?Bug
or feature ?
cheersMagnus
dependencies { groovy group: 'org.codehaus.groovy', name: 'groovy-all',
version: '1.8.3'
providedCompile 'javax.servlet:servlet-api:2.4' // consolidate logging
to use slf4 api and logback as implementation // Note ordering of
dependencies to ensure logback initialized properly as early as possible
compile 'ch.qos.logback:logback-core:0.9.29',
'ch.qos.logback:logback-classic:0.9.29',
'org.slf4j:jcl-over-slf4j:1.6.1', 'org.slf4j:jul-to-slf4j:1.6.1'
compile("org.grails:grails-gorm:$grailsVersion") { exclude
group: 'org.springframework', module: 'spring-webmvc' exclude group:
'org.springframework', module: 'spring-jms' } compile
"org.grails:grails-bootstrap:$grailsVersion",
"org.springframework:spring-beans:$springVersion",
"org.apache.tapestry:tapestry-core:$tapestryVersion",
"org.apache.tapestry:tapestry-spring:$tapestryVersion"
compile('org.got5:tapestry5-jquery:2.6.0') //{changing=true} compile
'joda-time:joda-time:1.6.2', 'joda-time:joda-time-hibernate:1.2'
compile 'com.h2database:h2:1.3.157' testCompile 'junit:junit:4.8.2',
'org.spockframework:spock-core:0.5-groovy-1.8@jar', /* NOTE I HAVE TO
DISABLE TRANSITIVE DEPS HERE in milestone 6 */
'org.spockframework:spock-spring:0.5-groovy-1.8@jar', /* NOTE I HAVE TO DISABLE
TRANSITIVE DEPS HERE in milestone 6 */
"org.grails:grails-test:$grailsVersion"}
Date: Sat, 5 Nov 2011 15:40:07 -0600
From: [email protected]
To: [email protected]
Subject: [gradle-dev] Dependency resolution improvements
G'day
Over the past couple of weeks we've made some significant improvements to the
way dependency resolution works in Gradle. There are some things that are just
refactorings, some performance improvements, and a couple of new features.
These changes should make it into M6, which is due out soon.
Caching Improvements:The big new change in M6 is that Gradle now takes control
of the caching of Dynamic Versions (eg 1.+) and Changing Modules (eg SNAPSHOT).
A new metadata file in the cache, "dynamic-versions.bin", caches information
about module resolution on a per-resolver basis.
Dynamic Versions: If a resolver tells us that 1.+ is actually 1.5, we remember
that for next time. That way, we can look directly in the module cache to get
the descriptor for 1.+, rather than first having to resolve 1.+ to 1.5.
Dynamic Version cache entries expire by default in 24 hours. This value can be
modified using configuration.resolutionStrategy.cacheDynamicVersionsFor 2,
"minutes"
Changing Modules:Similarly, if a dependency is flagged as changing, or a
resolver tells us the module is changing (eg SNAPSHOT), we remember when we
last fetched the module meta-data and artifacts. If we find a cache entry
indicating it's ok to use the cached version, we use them. Otherwise, we bypass
the cache and download the descriptor and artifacts again.
Changing Module cache entries expire by default in 24 hours. This value can be
modified using configuration.resolutionStrategy.cacheChangingModulesFor 2,
"minutes"
Maven Snapshots:By implementing these resolution caches ourselves, we have been
able to remove the previous snapshot ttl that was provided by the
WharfIbiblioResolver. This has greatly simplified things internally.
Offline Mode:Although nothing official has been added, we should probably try
out an offline switch in our build that sets these cache values very high. In
theory, no requests should be required when everything has be previously
retrieved.
Performance Improvements:Although the new caching improvements help a lot with
performance when things have been resolved before, we've also done some work to
improve the performance when resolving dependencies against maven repositories.
Namely:
- Don't request javadoc and source jars when resolving the POM- Don't request
.sha1 and .md5 files when not required- Don't use HTTP HEAD to check existence
of file prior to downloading them.
Together, these improvements have reduced the number of HTTP requests required
to resolve a static module from 11 down to 2, and a simple SNAPSHOT module from
25 requests down to 4!
Refactoring:
Adam spent a bunch of time replacing the core Ivy ResolveEngine with our own
implementation. This has already proven beneficial in making the other
improvements simpler.
For most of the internals, we've started using our own from-scratch
implementation of org.apache.ivy.plugins.resolver.DependencyResolver, in place
of extending AbstractResolver, BasicResolver, ChainResolver etc. Our own
implementations need only implement 3 key methods on the interface, and we've
removed a whole bunch of ivy black magic in the process.
Another improvement has been to remove some of the power of the low-level ivy
resolvers that perform the actual resolution. Previously, each resolver could
look at what had been resolved earlier in the chain, and determine if it do
anything or accept the earlier resolution. Now each resolver behaves
independently, with the high level UserResolverChain doing the work of choosing
the best resolved version; short-circuiting resolution when looking up static
versions.
---------
I've been using the latest code as my local gradle install and I'm pretty sure
I've ironed out most of the kinks for M6. Please give it a go and shout if
anything isn't behaving. Note that the artifact cache version is changed, so
you're initial builds will be slow due to the need to download everything again.
Enjoy!
--
Darrell (Daz) DeBoer
Principal Engineer, Gradleware
http://www.gradleware.com
