Hello Josh,
many thanks for your very detailed answer!
I will go through your lines.
On 24.11.2014 17:29, Josh Suereth wrote:
We've noticed similar for sbt. Specifically if your project has the
following characteristics:
- Large number of dependencies (the transitive closure of dependencies
numbering 500+ total)
this is the case. We depend on arquillian and all the stuff around
testing. So we can easily accumulate 150 libs or more per module/project.
- Large number of graph invalidations during resolution (i.e. library A
causes library B to get resolved again and again)
okay makes more sense now for me. Can I get rid of the mulitple resolves?
- A large number of configurations (most of our projects have 8-9
configurations, including through all the transitive dependencies).
Also here a full hit. We have 13 configurations (incl. master and
default). But not all of them are used in all modules/projects.
- Any amount of changing/snapshot dependencies.
we have changing snapshots only on two configurations. But we have them.
Basically every module/project provides a snapshot version of a jar,
javadoc-jar, source-jar, sometimes a war or ear file also
We're taking a super aggressive approach to solve this. You can read some
details here: http://www.scala-sbt.org/0.13/docs/Cached-Resolution.html
Looks interessting. Will go through it this week. Looks like you try to
do a full cached lookup. Would also be fine for me, if it boosts the
performance. But let me read it thouroughly. So I can repsond to it better.
A few notes on this approach and why we haven't (yet) shared with a larger
audience:
- It looks down the possible supported extensions in Ivy. E.g.
ConflictResolvers + DependencyMediators are no longer open. We only allow
a few known + supported mechanisms.
Ups, do not know both. Also here some homework for me. Could you point
me to the source/doc of it?
- It no longer directly mirrors Ivy resolution in certain cases.
Particularly, we deviate in how transitive forced dependencies are handled,
specifically as we believe the attempt to unify Maven-semantics in Ivy
using force has lead to a lot of user confusion around version-number
choices.
Sorry you lost me here. I do not get fully the point.
Have you customized in sbt the dependency lookup like IVY handles it?
And also do not get the point about Maven semantics vs Ivy. Also about
the issue "force".
So far, caching partial graphs in resolution has shown a TON of promise.
There's a lot of caveats and things to work out, and it's probably not
practical for Ivy to outlaw certain classes of extension it provides
currently, but I want to highly emphasize that if we manage to work out
issues, I'd love to see an Ivy 3.0 that has such caching baked in, and
makes the design trade-offs for super-fast resolution.
For me it would be okay to say:
ivy-org: "my.org" => uncached
* => cached
on staring ivy.
So in jenkins I can say always to resolve, if it takes some minutes it
is not nice but okay, it is not a human being waiting for it.
And for the developers, mostly they do not need to resolve new libs, so
in over 90% of the cases a fully cached * (excluding org="my.org") would
already be fine for a performance tuning.
This could be done e.g. by providing a JVM system property like
"--fullyCachedResolve=true --noCache=my.org" or something else.
I realize this isn't exactly a solution for ivy today, however things you
can do to improve resolution:
- Avoid conflicting versions in your transitive dependencies so there
are no graph invalidations. This can be done by fine-tuning which
versions you use. Check your ivy-xml reports for evictions and try to
reduce these.
Okay I will do the check with my other homework. thanks for the hint.
- Avoid dependencies. Try to limit the total number of dependencies
for any resolution to under 100 libraries.
That is not really possible. Since we do not have control over all other
transitive dependencies. Especially not if you work on wildfly and its
arquillian container :(
Hope that helps. We really love the ivy model, but we do feel the core
engine implementation is ripe for more optimisation. Hopefully this
mechanism of caching pans out long term not just for sbt, but also for Ivy.
It helped a lot, many thanks again.
I see the same issues with IVY. We decided against maven, since of it's
nature of forcing developers into the maven lifecycle. But right now I
am not sure if IVY (with our own build) was the better decission. Since
IVY eats up all productivity of the developers. They hate the words "do
a full build" since it takes more time as you can spend for going for a
coffee...
- Josh
On Mon, Nov 24, 2014 at 3:51 AM, hkais...@googlemail.com <hkais...@gmail.com
wrote:
