We've noticed similar for sbt. Specifically if your project has the following characteristics:
- Large number of dependencies (the transitive closure of dependencies numbering 500+ total) - Large number of graph invalidations during resolution (i.e. library A causes library B to get resolved again and again) - A large number of configurations (most of our projects have 8-9 configurations, including through all the transitive dependencies). - Any amount of changing/snapshot dependencies. We're taking a super aggressive approach to solve this. You can read some details here: http://www.scala-sbt.org/0.13/docs/Cached-Resolution.html A few notes on this approach and why we haven't (yet) shared with a larger audience: - It looks down the possible supported extensions in Ivy. E.g. ConflictResolvers + DependencyMediators are no longer open. We only allow a few known + supported mechanisms. - It no longer directly mirrors Ivy resolution in certain cases. Particularly, we deviate in how transitive forced dependencies are handled, specifically as we believe the attempt to unify Maven-semantics in Ivy using force has lead to a lot of user confusion around version-number choices. So far, caching partial graphs in resolution has shown a TON of promise. There's a lot of caveats and things to work out, and it's probably not practical for Ivy to outlaw certain classes of extension it provides currently, but I want to highly emphasize that if we manage to work out issues, I'd love to see an Ivy 3.0 that has such caching baked in, and makes the design trade-offs for super-fast resolution. I realize this isn't exactly a solution for ivy today, however things you can do to improve resolution: - Avoid conflicting versions in your transitive dependencies so there are no graph invalidations. This can be done by fine-tuning which versions you use. Check your ivy-xml reports for evictions and try to reduce these. - Avoid dependencies. Try to limit the total number of dependencies for any resolution to under 100 libraries. Hope that helps. We really love the ivy model, but we do feel the core engine implementation is ripe for more optimisation. Hopefully this mechanism of caching pans out long term not just for sbt, but also for Ivy. - Josh On Mon, Nov 24, 2014 at 3:51 AM, hkais...@googlemail.com <hkais...@gmail.com > wrote: > Hello together, > > I am facing the problem, that my IVY publish into a local file cache is > taking "ages". It takes at least 30 seconds to publish a set of max of 3 > artifacts. (jar, source, javadoc) > > Same issue is on IVY resolve. Here it seems to be the up to date check of > Ivy to verify if the resources are still the newest. > How can I modify the behaviour of this update check? > e.g. here it takes sometimes minutes, even if the stuff is already cached. > > Since we have a multiproject environment with 13 projects IVY eats up > following times: > 13x 30sec for local publish => 6,5 minutes > 13x at least 30sec for resolve => 6,5 minutes > > So our build is waiting about 10-13 minutes only for IVY. The whole build > (incl. IVY) takes about 15-16 minutes. > > Can someone help me out of the dark how to improve IVY publish and IVY > resolve for a local - very fast - build. > On our CI jenkins, it is okay - not nice but okay - to take what time. > > best wishes, > hkais >
