Yes, of course! That`s what I`m saying. Using different resolution caches on parallel builds works much better but it fails anyway, if there is being downloaded new artifact from repository (to default cache). So lockStrategy=artifact-lock doesn`t help.
Regards, Uldis, C. T. Co [email protected] -----Original Message----- From: Sam Berlin [mailto:[email protected]] Sent: piektdiena, 2009. gada 20. martā 18:05 To: [email protected] Subject: Re: Running Ivy in parallel builds Even with artifact locking enabled, the resolution cache *must* be unique per build if concurrent builds are occurring. Sam On Fri, Mar 20, 2009 at 11:50 AM, Uldis Karlovs-Karlovskis <[email protected]> wrote: > The parallel artifacts download unfortunately failed. Artifact-lock is not helping. However, this is much better than it was. > > Regards, > Uldis, C. T. Co > [email protected] > > > -----Original Message----- > From: Alan Chaney [mailto:[email protected]] > Sent: piektdiena, 2009. gada 20. martā 17:47 > To: [email protected] > Subject: Re: Running Ivy in parallel builds > > Hi Maarten > Maarten Coene wrote: >> Hi, >> >> I think you can solve it like this >> - use the same cache for your artifacts >> - use a different (per build?) cache for your resolution data >> - use the "artifact-lock" locking strategy on your artifact cache >> >> More info about the caches is here: >> http://ant.apache.org/ivy/history/latest-release/settings/caches.html >> >> Maarten >> >> > I hope you don't consider the following too off-topic but I'm trying to > set up a parallel build and I'm a little confused about the docs as they > relate to caching. In > http://ant.apache.org/ivy/history/latest-release/concept.html#cache > it states: > While there is always only one resolution cache, you can define multiple > repository caches > <http://ant.apache.org/ivy/history/latest-release/settings/caches.html>, > each resolver > <http://ant.apache.org/ivy/history/latest-release/settings/resolvers.html> > being able to use a separate cache. > > But in the actual caches/cache description it says that you can define > the cachedir within your ivysettings.xml. I assume that the 'only one > resolution cache' means 'per ivy settings'. Is this correct? > > If you have multiple ivy settings defined in a project and you specify > which settings you use you can presumably use different resolution > caches for different aspects of the project? > > Regards > > Alan Chaney > > > >> >> >> ----- Original Message ---- >> From: Uldis Karlovs-Karlovskis <[email protected]> >> To: [email protected] >> Sent: Friday, March 20, 2009 11:45:21 AM >> Subject: RE: Running Ivy in parallel builds >> >> OMG, this is totally unacceptable. We started to use Ivy to optimize space >> for builds and automatize build tools setups. My Ivy cache stores about >> 600Mb of different tools initially which are reused then for every build. So >> using cache per build is a nightmare! Using cache per running parallel build >> sounds better but developing this custom configuration can make a lot of >> unexpected problems afterthen. >> Ivy on his own should know if cache is used by another process and wait for >> it. Does anybody is going to implement this functionality? Since I`m using >> special Ivy wrapper I can try to do it on my own on my side but it is not a >> good idea anyway. >> >> Regards, >> Uldis, C. T. Co >> [email protected] >> >> >> -----Original Message----- >> From: Martin Eigenbrodt [mailto:[email protected]] >> Sent: ceturtdiena, 2009. gada 19. marta 20:31 >> To: [email protected] >> Subject: Re: Running Ivy in parallel builds >> >> I do run Ivy on Hudson and use one Cache per "Executor". So builds in >> parallel have different caches but sequential builds may share >> a cache. Hudson provides an Environment Variabel with the Executor Number >> that I use within the ivysettings.xml to set different Cachedirs. A similair >> setup may be possible in other environments. >> A second concurency issue is that the buildnumber and deploy tasks are >> seperated. There should be an atomic "deployWithNext Buildnumber" or you >> will run in problems when the same project is build in parallel. >> >> Best regards, >> >> Martin2009/3/19 Sam Berlin <[email protected]> >> >> >>> Yes, I have noticed significant problems when using ivy in a parallel >>> build environment (essentially, any build-server), and we've only had >>> Ivy in use on the buildserver for ~12 hours. The only solution I can >>> think of is to make the cache local per build, which is pretty far >>> from ideal. >>> >>> Any suggestions for how to make Ivy more robust against concurrent builds? >>> >>> Sam >>> >>> 2009/3/19 Uldis Karlovs-Karlovskis <[email protected]>: >>> >>>> And this is not all. Previous case was when artifact is already in >>>> >> cache. >> >>> If >>> >>>> Ivy needs to download new artifact then parallel execution gives >>>> >>> something >>> >>>> like this: >>>> >>>> --------------------------------------------------------------------- >>>> | | modules || artifacts | >>>> | conf | number| search|dwnlded|evicted|| number|dwnlded| >>>> >>>> --------------------------------------------------------------------- >>>> | default | 4 | 0 | 0 | 0 || 6 | 0 | >>>> >>>> --------------------------------------------------------------------- >>>> >>>> :: problems summary :: >>>> :::: WARNINGS >>>> problem while downloading module descriptor: >>>> >>>> >> http://server/download?name=somemodule&version=1.3.1.16&file=somemodule&file >> >>>> Type=xml: impossible to move part file to definitive one: >>>> /root/.ivy2/cache/ctco/somemodule/ivy-1.3.1.16.xml.original.part -> >>>> /root/.ivy2/cache/ctco/somemodule/ivy-1.3.1.16.xml.original (21ms) >>>> >>>> module not found: ctco#somemodule;1.3.1.16 >>>> >>>> ==== bones-repository: tried >>>> >>>> >>>> >>>> >> http://server/download?name=somemodule&version=1.3.1.16&file=somemodule&file >> >>>> Type=xml >>>> >>>> :::::::::::::::::::::::::::::::::::::::::::::: >>>> >>>> :: UNRESOLVED DEPENDENCIES :: >>>> >>>> :::::::::::::::::::::::::::::::::::::::::::::: >>>> >>>> :: ctco#somemodule;1.3.1.16: not found >>>> >>>> :::::::::::::::::::::::::::::::::::::::::::::: >>>> >>>> >>>> >>>> :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS >>>> command exit code: 1 >>>> >>>> Regards, >>>> Uldis, C. T. Co >>>> [email protected] >>>> >>>> >>>> -----Original Message----- >>>> From: Uldis Karlovs-Karlovskis [mailto:[email protected]] >>>> Sent: ceturtdiena, 2009. gada 19. marta 17:37 >>>> To: [email protected] >>>> Subject: Running Ivy in parallel builds >>>> >>>> Hello! >>>> >>>> >>>> >>>> Have You tested running Ivy in more than one build at the time? >>>> >>>> I see there are few files in cache which got updated every time I run >>>> >>> Ivy. >>> >>>> When my build system is running more than one process at the time >>>> >>> sometimes >>> >>>> I got this exception: >>>> >>>> :: retrieving :: lv.ctco#scm-bootstrap >>>> >>>> confs: [default] >>>> >>>> Caught: java.lang.RuntimeException: problem during retrieve of >>>> <namespace>#<module>: java.text.ParseException: failed to parse report: >>>> /root/.ivy2/cache/lv.ctco-scm-bootstrap-default.xml: Premature end of >>>> >>> file. >>> >>>> command exit code: 1 >>>> >>>> >>>> >>>> >>>> >>>> Uldis Karlovs - Karlovskis >>>> >>>> Lead SCM Engineer >>>> >>>> System Configuration Management >>>> >>>> C.T.Co >>>> >>>> Mobile: +371 29 345 210 >>>> >>>> >>>> >>>> >>>> >>>> >> >> >> >> >> >> !DSPAM:49c3796450102136417547! >> >> > > > >
