On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mgit...@gmail.com> wrote:
> As you describe, the immediate problem is that, with settings, the
> EXECUTOR_NUMBER environment variable is not being picked up. The following
> line is really just establishing a default:
> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
That's correct. The environment variable EXECUTOR_NUMBER is set by Jenkins, If
you're not running on Jenkins, then, EXECUTOR_NUMBER isn't set. I use that line
to set it if it's not already set.
> For debugging's sake, I'd try leaving that out. The executor number should
> be required.
>
> For debugging's sake, I'd also try outputting the value in the Ant build
> just prior to calling ivy:settings.
>
> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.
All three of these are true. In fact, when I first used my ivysettings.xml
file, I set the executor number manually in my environment. When I did the
build, the cache directory was being set correctly. At least when the
<ivy:resolve> took place, and the correct cache was being created. I also made
sure that the cache was set to $HOME/.ivy2/cache-o when there was no
EXECUTOR_NUMBER environment variable. To me, this was the default way the user
would run it. Everything looked great.
Suddenly builds on Jenkins started to fail and I figured that happened when
parallel builds took place and were wiping out each other's cache. Printing a
few debug statements before and after the <ivy:settings> task led me to realize
that the cache wasn't being set. That's when I read through the <ivy:settings>
task documentation and discovered that <ivy:configure> did more or less the
same thing, but that the <ivy:configure> task did the configuration immediately
instead of "when needed". Switching to <ivy:configure> fixed the issue.
> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.
It is. My Ivy project is a subdirectory to the user's project. The developer
puts in the lines "<property name="ivy.dir" value="${basedir}/ivy.dir"/> and
<import file="${ivy.dir}/ivy.tasks.xml"/> into their build. The "ivy.task.xml"
file calls the "<ivy:settings>" task for the user. It runs before any other
task is called. The user calls <ivy:cleancache> this way:
<property name="ivy.cleancache" value="true"/>
<target name="clean">
<if>
<istrue value="${ivy.cleancache}"/>
<then>
<ivy:cleancache/>
</then>
</if>
<delete>
<fileset dir="${target.dir}"/>
</fileset>
</target>
The user can override the value of ${ivy.cleancache} via a build.properties
file, so the cache isn't cleaned every time the developer runs the "clean"
target. Thus, <ivy:settings/> is called first before <ivy:cleancache/>.
> It appears that your resolution cache is not being defined on a
> per-executor basis. If each executor build has its own target directory,
> that's OK. But then, why not do that for both your resolution cache and
> your repository cache? So you'd specify resolutionCacheDir and
> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
The resolution cache needs to be defined so that parallel builds don't
interfere with each other. There can also be an issue of a developer using
multiple working directories of the same project, and this is also an issue
with Jenkins too. The resolution cache gets built pretty quickly, so if you
delete it and it has to be built from scratch, it takes maybe another second to
do. Thus, putting the resolution cache per checkout resolves the issue of
multiple builds sharing the same resolution cache since each will have its own.
When the developer does a clean, they delete the "target" directory and the
resolution cache. No real problem since it can be quickly rebuilt.
The Ivy cache is a different issue. If this cache is deleted, all the jars must
be redownloaded which takes time. Besides if project A downloads version 1.3 of
foo.jar, and project B also needs that jar, project B doesn't have to download
it. That's the whole purpose of the Ivy cache. Thus, the resolution cache is
placed per project while the Ivy jar cache is shared between different projects.
> I forget if Jenkins checks out the source into a different location for
> each executor build (it should), but if that's the case and you're sending
> your output to a location relative to your source, then establishing a
> relative path to your caches should suffice, and there's no need to
> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now that I
> think of it, specifying Ivy caches for a CI build that are based on
> ivy.default.ivy.user.dir, considering that its default is user.home/.ivy2/,
> is a bit of an antipattern.
No, each job has an established working directory and it uses the same working
directory no matter which executor is running. This makes sense since the first
time a job might run, it runs on executor #1 and the second time on executor
#0. If there were separate working directories per executor, you couldn't take
advantage of a version control update which only has to copy a few files over
instead of a whole new working directory.
The "cache" really doesn't exist on Jenkins because I wipe it out for each
build. That's why each executor needs a separate build. I could just as easily
put that under the "target" directory like I do the resolution cache.
Unfortunately, the same build.xml is also used by the developer. Their machines
are slower and on a slower network connection than the servers. Jenkins can
download all of the jars in seconds. On the developer's machines, it can take
minutes. Thus, the cache is very important to the developers.
This is the problem: Both developers and Jenkins use the same build process,
and the same build files. But, both use the Ivy cache very differently.
Developers want it to act like a cache: Saving them valuable time by not having
to redownload files over and over. Jenkins wants a clean slate to make sure
that there isn't a jar issue during its build process. Satisfying these two
completely different requirements while using the same build process and files
lead me to this design.
On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mgit...@gmail.com> wrote:
> Let me take a stab at addressing your immediate problem, even though I
> don't have a direct answer. But then I'm going to suggest a different
> approach that avoids that problem.
>
> Frankly, I've never used configure, only settings. The configure
> documentation explains the primary difference (basically settings is lazy):
> http://ant.apache.org/ivy/history/latest-milestone/use/configure.html
>
> As you describe, the immediate problem is that, with settings, the
> EXECUTOR_NUMBER environment variable is not being picked up. The following
> line is really just establishing a default:
> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>
> For debugging's sake, I'd try leaving that out. The executor number should
> be required.
>
> For debugging's sake, I'd also try outputting the value in the Ant build
> just prior to calling ivy:settings.
>
> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.
>
> And there's something I find odd about your caches specification:
> <caches
> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
> resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
>
> It appears that your resolution cache is not being defined on a
> per-executor basis. If each executor build has its own target directory,
> that's OK. But then, why not do that for both your resolution cache and
> your repository cache? So you'd specify resolutionCacheDir and
> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
>
> I forget if Jenkins checks out the source into a different location for
> each executor build (it should), but if that's the case and you're sending
> your output to a location relative to your source, then establishing a
> relative path to your caches should suffice, and there's no need to
> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now that I
> think of it, specifying Ivy caches for a CI build that are based on
> ivy.default.ivy.user.dir, considering that its default is user.home/.ivy2/,
> is a bit of an antipattern.
>
> On Thu, Nov 29, 2012 at 8:35 AM, David Weintraub <qazw...@gmail.com> wrote:
>
>> When is it better to use <ivy:settings> vs. <ivy:configure>?
>>
>> I ask because I was having a problem with my Ivy setup. We use Subversion
>> and have multiple projects setup that use Ant. In order to centralize our
>> jar dependencies, I've decided to implement Ivy into our process.
>>
>> I did this by creating an Ivy project (https://github.com/qazwart/ivy.dir)
>> in Subversion. Users merely set `svn:externals` to include this project,
>> make some minor changes in their build.xml, and add in a "ivy.xml" file,
>> and everything is set. I control the ivysettings.xml file, so I point Ivy
>> to our repository. The Ivy jar is part of the Ivy project, so users don't
>> have to install it. The idea was to make Ivy integration as painless as
>> possible.
>>
>> We use Jenkins as our build system, and I decided it would be good if we
>> could clean the Ivy cache before each build. However, out Jenkins system
>> has six executors, so as many as six builds could happen at once. Cleaning
>> the cache in one build while another one runs could cause problems, so I
>> modified my ivysettings.xml file to use a different cache depending upon
>> the executor:
>>
>> <ivysettings>
>> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>> <caches
>>
>> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
>> resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
>> <settings defaultResolver="default"/>
>> <include file="${ivy.dir}/ivysettings-public.xml"/>
>> <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
>> <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
>> <include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
>> <include
>> url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
>> </ivysettings>
>>
>> The problem is that if I did an <ivy:cleancache>, the cache ID was always
>> set to "cache-0". When I switched from "<ivy:settings>" to
>> "<ivy:configure>", the problem went away. (In the standard build process,
>> the <ivy:cleancache> task is executed before the <ivy:resolve> task).
>>
>> I'm told that "<ivy:settings>" can do multiple configurations, but it looks
>> like <ivy:configure> also can do multiple settings too. I also read that
>> "<ivy:configure>" was deprecated in some mailings, but it doesn't state
>> that in the on line documentation.
>>
>> So what is the difference between <ivy:settings> and <ivy:configure>, and
>> when should I use one over the other?
>>
>> --
>> David Weintraub
>> qazw...@gmail.com
>>
