RE: Difficult with concurrent builds

2015-04-07 Thread Loren Kratzke
Your comment about Ivy not always delivering the latest artifact caught my 
attention. That in combination with perceived cache issues makes me just shake 
my head.

Deleting cache directories is only treating the symptoms. When you think about 
it, what is the point of a cache if you delete it? Furthermore, you consider 
the cache poison to the build because it is. It's a bad place to be. I have 
been there my friend.

See my recommendations on how to take absolute control of your dependency 
resolution and retrieval strategies in Ivy. Use it, like it, then scream it 
from the mountain tops until it becomes documentation on the site. It's good 
stuff.

http://mail-archives.apache.org/mod_mbox/ant-ivy-user/201504.mbox/%3C2ffef9f4335143be92cd684f2199808c%40MAIL02.blueorigin.com%3E

L.K.

-Original Message-
From: Qazwart [mailto:qazw...@gmail.com] 
Sent: Tuesday, April 07, 2015 8:48 AM
To: ivy-user@ant.apache.org
Subject: Re: Difficult with concurrent builds

Here's how I fixed the ivy cache issue with Jenkins:

In my ivysettings.xml:

ivysettings
property name=env.EXECUTOR_NUMBER value=0 override=false/
caches

defaultCacheDir=${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}
resolutionCacheDir=${ivy.dir}/../target/ivy.cache/
settings defaultResolver=default/
include file=${ivy.dir}/ivysettings-public.xml/
include url=${ivy.default.settings.dir}/ivysettings-shared.xml/
include url=${ivy.default.settings.dir}/ivysettings-local.xml/
include url=${ivy.default.settings.dir}/ivysettings-main-chain.xml/
include url=${ivy.default.settings.dir}/ivysettings-default-chain.xml/
/ivysettings

Jenkins sets the EXECUTOR_NUMBER environment variable depending on which 
executor is used. If the build is run locally, the default cache will be  
$HOME/.ivy2/cache-0.

I ran into this cache issue because we clean out the cache from our locally 
built jars with each build. This guarantees we're working with the latest 
version of those jars (something that Ivy didn't always do all the time). 
Deleting jars while an Ivy build is running doesn't make Jenkins very happy. It 
took us a while to realize why builds were failing.

--
David Weintraub
qazw...@gmail.com

perl -e 'print Just another second rate Perl Hacker\n;'

 On Apr 7, 2015, at 9:58 AM, Roberto C. Sánchez robe...@connexer.com wrote:
 
 On Tue, Apr 07, 2015 at 09:53:08AM +0200, Geißler, Daniel wrote:
 Hi Roberto,
 
 did you have a look into locking strategies, as far as i know ivy does no 
 locking by default wich may lead to your problems:
 
 http://ant.apache.org/ivy/history/latest-milestone/settings/lock-stra
 tegies.html
 
 Hi Daniel,
 
 Thank you for responding.  This appears to be a low traffic list :-)
 
 I did not realize that multiple builds sharing the cache could present 
 a problem.  In any event, each of my builds is configured to use its 
 own Ivy cache directory based on the Jenkins workspace directory.  I 
 don't think that affects the part of the build with which I am 
 currently experiencing a problem.  Additionally, the documentation 
 link you provided does not give an example of how I would use the 
 locking strategy.
 
 I think that the problem has something to do with the deliver/publish 
 logic, but I cannot seem to figure out precisely what or where.  I 
 have not observed any problem with any of my downloaded artifacts for 
 the dependencies.  I have only seen a problem with the 
 delivered/published ivy.xml pertaining to the project being built at 
 the same time as it is being built by another process (but on a different 
 branch).
 
 Regards,
 
 -Roberto
 --
 Roberto C. Sánchez
 http://people.connexer.com/~roberto
 http://www.connexer.com



Re: Difficult with concurrent builds

2015-04-07 Thread Qazwart
Here’s how I fixed the ivy cache issue with Jenkins:

In my ivysettings.xml:

ivysettings
property name=env.EXECUTOR_NUMBER value=0 override=false/
caches

defaultCacheDir=${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}
resolutionCacheDir=${ivy.dir}/../target/ivy.cache/
settings defaultResolver=default/
include file=${ivy.dir}/ivysettings-public.xml/
include url=${ivy.default.settings.dir}/ivysettings-shared.xml/
include url=${ivy.default.settings.dir}/ivysettings-local.xml/
include url=${ivy.default.settings.dir}/ivysettings-main-chain.xml/
include url=${ivy.default.settings.dir}/ivysettings-default-chain.xml/
/ivysettings

Jenkins sets the “EXECUTOR_NUMBER” environment variable depending on which 
executor is used. If the build is run locally, the default cache will be  
$HOME/.ivy2/cache-0.

I ran into this cache issue because we clean out the cache from our locally 
built jars with each build. This guarantees we’re working with the latest 
version of those jars (something that Ivy didn’t always do all the time). 
Deleting jars while an Ivy build is running doesn’t make Jenkins very happy. It 
took us a while to realize why builds were failing.

--
David Weintraub
qazw...@gmail.com

perl -e 'print Just another second rate Perl Hacker\n;'

 On Apr 7, 2015, at 9:58 AM, Roberto C. Sánchez robe...@connexer.com wrote:
 
 On Tue, Apr 07, 2015 at 09:53:08AM +0200, Geißler, Daniel wrote:
 Hi Roberto,
 
 did you have a look into locking strategies, as far as i know ivy does no 
 locking by default wich may lead to your problems:
 
 http://ant.apache.org/ivy/history/latest-milestone/settings/lock-strategies.html
 
 Hi Daniel,
 
 Thank you for responding.  This appears to be a low traffic list :-)
 
 I did not realize that multiple builds sharing the cache could present a
 problem.  In any event, each of my builds is configured to use its own
 Ivy cache directory based on the Jenkins workspace directory.  I don't
 think that affects the part of the build with which I am currently
 experiencing a problem.  Additionally, the documentation link you
 provided does not give an example of how I would use the locking
 strategy.
 
 I think that the problem has something to do with the deliver/publish
 logic, but I cannot seem to figure out precisely what or where.  I have
 not observed any problem with any of my downloaded artifacts for the
 dependencies.  I have only seen a problem with the delivered/published
 ivy.xml pertaining to the project being built at the same time as it is
 being built by another process (but on a different branch).
 
 Regards,
 
 -Roberto
 -- 
 Roberto C. Sánchez
 http://people.connexer.com/~roberto
 http://www.connexer.com