RE: Difficult with concurrent builds
Your comment about Ivy not always delivering the latest artifact caught my
attention. That in combination with perceived cache issues makes me just shake
my head.
Deleting cache directories is only treating the symptoms. When you think about
it, what is the point of a cache if you delete it? Furthermore, you consider
the cache poison to the build because it is. It's a bad place to be. I have
been there my friend.
See my recommendations on how to take absolute control of your dependency
resolution and retrieval strategies in Ivy. Use it, like it, then scream it
from the mountain tops until it becomes documentation on the site. It's good
stuff.
http://mail-archives.apache.org/mod_mbox/ant-ivy-user/201504.mbox/%3C2ffef9f4335143be92cd684f2199808c%40MAIL02.blueorigin.com%3E
L.K.
-Original Message-
From: Qazwart [mailto:qazw...@gmail.com]
Sent: Tuesday, April 07, 2015 8:48 AM
To: ivy-user@ant.apache.org
Subject: Re: Difficult with concurrent builds
Here's how I fixed the ivy cache issue with Jenkins:
In my ivysettings.xml:
ivysettings
property name=env.EXECUTOR_NUMBER value=0 override=false/
caches
defaultCacheDir=${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}
resolutionCacheDir=${ivy.dir}/../target/ivy.cache/
settings defaultResolver=default/
include file=${ivy.dir}/ivysettings-public.xml/
include url=${ivy.default.settings.dir}/ivysettings-shared.xml/
include url=${ivy.default.settings.dir}/ivysettings-local.xml/
include url=${ivy.default.settings.dir}/ivysettings-main-chain.xml/
include url=${ivy.default.settings.dir}/ivysettings-default-chain.xml/
/ivysettings
Jenkins sets the EXECUTOR_NUMBER environment variable depending on which
executor is used. If the build is run locally, the default cache will be
$HOME/.ivy2/cache-0.
I ran into this cache issue because we clean out the cache from our locally
built jars with each build. This guarantees we're working with the latest
version of those jars (something that Ivy didn't always do all the time).
Deleting jars while an Ivy build is running doesn't make Jenkins very happy. It
took us a while to realize why builds were failing.
--
David Weintraub
qazw...@gmail.com
perl -e 'print Just another second rate Perl Hacker\n;'
On Apr 7, 2015, at 9:58 AM, Roberto C. Sánchez robe...@connexer.com wrote:
On Tue, Apr 07, 2015 at 09:53:08AM +0200, Geißler, Daniel wrote:
Hi Roberto,
did you have a look into locking strategies, as far as i know ivy does no
locking by default wich may lead to your problems:
http://ant.apache.org/ivy/history/latest-milestone/settings/lock-stra
tegies.html
Hi Daniel,
Thank you for responding. This appears to be a low traffic list :-)
I did not realize that multiple builds sharing the cache could present
a problem. In any event, each of my builds is configured to use its
own Ivy cache directory based on the Jenkins workspace directory. I
don't think that affects the part of the build with which I am
currently experiencing a problem. Additionally, the documentation
link you provided does not give an example of how I would use the
locking strategy.
I think that the problem has something to do with the deliver/publish
logic, but I cannot seem to figure out precisely what or where. I
have not observed any problem with any of my downloaded artifacts for
the dependencies. I have only seen a problem with the
delivered/published ivy.xml pertaining to the project being built at
the same time as it is being built by another process (but on a different
branch).
Regards,
-Roberto
--
Roberto C. Sánchez
http://people.connexer.com/~roberto
http://www.connexer.com
Re: Difficult with concurrent builds
Here’s how I fixed the ivy cache issue with Jenkins:
In my ivysettings.xml:
ivysettings
property name=env.EXECUTOR_NUMBER value=0 override=false/
caches
defaultCacheDir=${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}
resolutionCacheDir=${ivy.dir}/../target/ivy.cache/
settings defaultResolver=default/
include file=${ivy.dir}/ivysettings-public.xml/
include url=${ivy.default.settings.dir}/ivysettings-shared.xml/
include url=${ivy.default.settings.dir}/ivysettings-local.xml/
include url=${ivy.default.settings.dir}/ivysettings-main-chain.xml/
include url=${ivy.default.settings.dir}/ivysettings-default-chain.xml/
/ivysettings
Jenkins sets the “EXECUTOR_NUMBER” environment variable depending on which
executor is used. If the build is run locally, the default cache will be
$HOME/.ivy2/cache-0.
I ran into this cache issue because we clean out the cache from our locally
built jars with each build. This guarantees we’re working with the latest
version of those jars (something that Ivy didn’t always do all the time).
Deleting jars while an Ivy build is running doesn’t make Jenkins very happy. It
took us a while to realize why builds were failing.
--
David Weintraub
qazw...@gmail.com
perl -e 'print Just another second rate Perl Hacker\n;'
On Apr 7, 2015, at 9:58 AM, Roberto C. Sánchez robe...@connexer.com wrote:
On Tue, Apr 07, 2015 at 09:53:08AM +0200, Geißler, Daniel wrote:
Hi Roberto,
did you have a look into locking strategies, as far as i know ivy does no
locking by default wich may lead to your problems:
http://ant.apache.org/ivy/history/latest-milestone/settings/lock-strategies.html
Hi Daniel,
Thank you for responding. This appears to be a low traffic list :-)
I did not realize that multiple builds sharing the cache could present a
problem. In any event, each of my builds is configured to use its own
Ivy cache directory based on the Jenkins workspace directory. I don't
think that affects the part of the build with which I am currently
experiencing a problem. Additionally, the documentation link you
provided does not give an example of how I would use the locking
strategy.
I think that the problem has something to do with the deliver/publish
logic, but I cannot seem to figure out precisely what or where. I have
not observed any problem with any of my downloaded artifacts for the
dependencies. I have only seen a problem with the delivered/published
ivy.xml pertaining to the project being built at the same time as it is
being built by another process (but on a different branch).
Regards,
-Roberto
--
Roberto C. Sánchez
http://people.connexer.com/~roberto
http://www.connexer.com
Difficult with concurrent builds
I have a project using Ivy which is built via Jenkins. I have two
branches for this project (a stable branch and a current development
branch, which is the master branch in Git).
In Jenkins I have one build configured for each branch (I don't want to
mix the build histories). I have a post-commit hook in my Git
repository which checks for the branch(es) of the commit and then uses
to wget to retrieve the URL that triggers the build that corresponds to
the branch of the commit.
The problem that I am experiencing has to do with making changes in
quick succession on the two branches. My Ant build.xml does the normal
build routine and at the end publishes the artifacts into my own local
Ivy repository. When the builds run concurrently (because, for example,
I make a commit one branch and then immediately cherry-pick the commit
to the other branch, the build that finshes last uses the ivy.xml
desriptor from the build that finishes first. This is very frustrating
as it breaks all sorts of things. I can go into Jenkins and force a
rebuild of the one that finished last and everything turns out
correctly, with the correct descriptor being published the second time.
I looked through the Ivy source code and found a reference to a delivery
list file that looks in ${java.io.tmpdir}/delivery.properties. I tried
explicitly setting the ivy.delivery.list.file using Ant's tempfile/
task to ensure that concurrent builds won't accidentally touch the same
file. I was able to confirm that instead of /tmp/delivery.properties
that the Ivy publish task was using the randomly generated file name Ant
put into the ivy.delivery.list.file property.
Even with that, though, I am still seeing the buggy behavior. That is,
the build that finishes last uses the same descriptor as the build that
finishes first (line for line identical except for the revision and
publication attributes of the info tag.
Any help on this would be appreciated.
Regards,
-Roberto
--
Roberto C. Sánchez
http://people.connexer.com/~roberto
http://www.connexer.com
