Hi Maarten
Maarten Coene wrote:
Hi,
I think you can solve it like this
- use the same cache for your artifacts
- use a different (per build?) cache for your resolution data
- use the "artifact-lock" locking strategy on your artifact cache
More info about the caches is here:
http://ant.apache.org/ivy/history/latest-release/settings/caches.html
Maarten
I hope you don't consider the following too off-topic but I'm trying to
set up a parallel build and I'm a little confused about the docs as they
relate to caching. In
http://ant.apache.org/ivy/history/latest-release/concept.html#cache
it states:
While there is always only one resolution cache, you can define multiple
repository caches
<http://ant.apache.org/ivy/history/latest-release/settings/caches.html>,
each resolver
<http://ant.apache.org/ivy/history/latest-release/settings/resolvers.html>
being able to use a separate cache.
But in the actual caches/cache description it says that you can define
the cachedir within your ivysettings.xml. I assume that the 'only one
resolution cache' means 'per ivy settings'. Is this correct?
If you have multiple ivy settings defined in a project and you specify
which settings you use you can presumably use different resolution
caches for different aspects of the project?
Regards
Alan Chaney
----- Original Message ----
From: Uldis Karlovs-Karlovskis <[email protected]>
To: [email protected]
Sent: Friday, March 20, 2009 11:45:21 AM
Subject: RE: Running Ivy in parallel builds
OMG, this is totally unacceptable. We started to use Ivy to optimize space
for builds and automatize build tools setups. My Ivy cache stores about
600Mb of different tools initially which are reused then for every build. So
using cache per build is a nightmare! Using cache per running parallel build
sounds better but developing this custom configuration can make a lot of
unexpected problems afterthen.
Ivy on his own should know if cache is used by another process and wait for
it. Does anybody is going to implement this functionality? Since I`m using
special Ivy wrapper I can try to do it on my own on my side but it is not a
good idea anyway.
Regards,
Uldis, C. T. Co
[email protected]
-----Original Message-----
From: Martin Eigenbrodt [mailto:[email protected]]
Sent: ceturtdiena, 2009. gada 19. martā 20:31
To: [email protected]
Subject: Re: Running Ivy in parallel builds
I do run Ivy on Hudson and use one Cache per "Executor". So builds in
parallel have different caches but sequential builds may share
a cache. Hudson provides an Environment Variabel with the Executor Number
that I use within the ivysettings.xml to set different Cachedirs. A similair
setup may be possible in other environments.
A second concurency issue is that the buildnumber and deploy tasks are
seperated. There should be an atomic "deployWithNext Buildnumber" or you
will run in problems when the same project is build in parallel.
Best regards,
Martin2009/3/19 Sam Berlin <[email protected]>
Yes, I have noticed significant problems when using ivy in a parallel
build environment (essentially, any build-server), and we've only had
Ivy in use on the buildserver for ~12 hours. The only solution I can
think of is to make the cache local per build, which is pretty far
from ideal.
Any suggestions for how to make Ivy more robust against concurrent builds?
Sam
2009/3/19 Uldis Karlovs-Karlovskis <[email protected]>:
And this is not all. Previous case was when artifact is already in
cache.
If
Ivy needs to download new artifact then parallel execution gives
something
like this:
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 4 | 0 | 0 | 0 || 6 | 0 |
---------------------------------------------------------------------
:: problems summary ::
:::: WARNINGS
problem while downloading module descriptor:
http://server/download?name=somemodule&version=1.3.1.16&file=somemodule&file
Type=xml: impossible to move part file to definitive one:
/root/.ivy2/cache/ctco/somemodule/ivy-1.3.1.16.xml.original.part ->
/root/.ivy2/cache/ctco/somemodule/ivy-1.3.1.16.xml.original (21ms)
module not found: ctco#somemodule;1.3.1.16
==== bones-repository: tried
http://server/download?name=somemodule&version=1.3.1.16&file=somemodule&file
Type=xml
::::::::::::::::::::::::::::::::::::::::::::::
:: UNRESOLVED DEPENDENCIES ::
::::::::::::::::::::::::::::::::::::::::::::::
:: ctco#somemodule;1.3.1.16: not found
::::::::::::::::::::::::::::::::::::::::::::::
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
command exit code: 1
Regards,
Uldis, C. T. Co
[email protected]
-----Original Message-----
From: Uldis Karlovs-Karlovskis [mailto:[email protected]]
Sent: ceturtdiena, 2009. gada 19. martā 17:37
To: [email protected]
Subject: Running Ivy in parallel builds
Hello!
Have You tested running Ivy in more than one build at the time?
I see there are few files in cache which got updated every time I run
Ivy.
When my build system is running more than one process at the time
sometimes
I got this exception:
:: retrieving :: lv.ctco#scm-bootstrap
confs: [default]
Caught: java.lang.RuntimeException: problem during retrieve of
<namespace>#<module>: java.text.ParseException: failed to parse report:
/root/.ivy2/cache/lv.ctco-scm-bootstrap-default.xml: Premature end of
file.
command exit code: 1
Uldis Karlovs - Karlovskis
Lead SCM Engineer
System Configuration Management
C.T.Co
Mobile: +371 29 345 210
!DSPAM:49c3796450102136417547!
