> The essence of the "correct" solution is to control the location of the
> parent Ivy user directory, which by default is user.home/.ivy2/. Developers
> are OK to use that default. For the Jenkins build, you want to override
> that in the user properties that the Jenkins job passes to Ant. The
> property you want to set is ivy.default.ivy.user.dir, and I believe you
> want to set it to the Jenkins-specific WORKSPACE variable. From here, you
> can define the caches relative to ${ivy.default.ivy.user.dir}.
That actually makes a bit more sense. This way, I don't have to go through the
rigamarole of checking the value of the ivy.cleancache property I go through
when I do a clean, and I don't have to depend upon the developers putting the
<ivy:cleancache> in their build.xml.
I'm going to see if I can set this property in my ivy.tasks.xml file, so I
don't have to redefine all of my Jenkins jobs. I'll also have to see if this
solves the <ivy:configuration> vs. the <ivy:setitngs> issue which caused this
whole thing to come up in the first place.
Thank you for your answer.
On Dec 1, 2012, at 12:04 PM, Mitch Gitman <mgit...@gmail.com> wrote:
> I'm a bit embarrassed by the answer I gave you because it was as recently
> as April that I was actively supporting a Jenkins cluster, and in that
> short time my Jenkins knowledge has slipped a bit. Since I'm going to have
> to do another Jenkins setup myself in about a week, I decided to refresh
> myself a bit about what I'd done before.
>
> I didn't get very far, but the most important bit of advice I can give you,
> even if it might not be what you want to hear, is this: don't be mapping
> Ivy caches to Jenkins executors.
>
> The essence of the "correct" solution is to control the location of the
> parent Ivy user directory, which by default is user.home/.ivy2/. Developers
> are OK to use that default. For the Jenkins build, you want to override
> that in the user properties that the Jenkins job passes to Ant. The
> property you want to set is ivy.default.ivy.user.dir, and I believe you
> want to set it to the Jenkins-specific WORKSPACE variable. From here, you
> can define the caches relative to ${ivy.default.ivy.user.dir}.
>
> This solves your problem: "This is the problem: Both developers and Jenkins
> use the same build process, and the same build files." The cache will be
> preserved for developers—and trusted for all modules except those that are
> publishing continually changing integration versions. The cache will be
> wiped out every time on the CI server because you've told the CI build
> (unlike the dev build) to run ivy:cleancache, but this is fine because for
> a CI build there's a greater premium on pristineness and reproducibility
> than speed. Because each Jenkins job has its own dedicated workspace and no
> two executors will be running the same job simultaneously, you're assured
> that there will be no concurrency issues between parallel builds.
>
> One clarification about terminology. The two types of cache are the
> resolution cache and the repository cache.
>
> If you're not satisfied about the Jenkins build wiping out the repository
> cache every time, there are steps you can take. But get the safe thing
> working first and optimize later. And even then, you should only be
> creating a cache and trusting it for those dependencies (like third-party
> libraries) that never change within the same version.
>
> On Thu, Nov 29, 2012 at 11:18 PM, David Weintraub <qazw...@gmail.com> wrote:
>
>> On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mgit...@gmail.com> wrote:
>>> As you describe, the immediate problem is that, with settings, the
>>> EXECUTOR_NUMBER environment variable is not being picked up. The
>> following
>>> line is really just establishing a default:
>>> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>>
>> That's correct. The environment variable EXECUTOR_NUMBER is set by
>> Jenkins, If you're not running on Jenkins, then, EXECUTOR_NUMBER isn't set.
>> I use that line to set it if it's not already set.
>>
>>> For debugging's sake, I'd try leaving that out. The executor number
>> should
>>> be required.
>>>
>>> For debugging's sake, I'd also try outputting the value in the Ant build
>>> just prior to calling ivy:settings.
>>>
>>> Also, I'd make sure that ivy:settings is always getting called before
>>> ivy:cleancache.
>>
>> All three of these are true. In fact, when I first used my ivysettings.xml
>> file, I set the executor number manually in my environment. When I did the
>> build, the cache directory was being set correctly. At least when the
>> <ivy:resolve> took place, and the correct cache was being created. I also
>> made sure that the cache was set to $HOME/.ivy2/cache-o when there was no
>> EXECUTOR_NUMBER environment variable. To me, this was the default way the
>> user would run it. Everything looked great.
>>
>> Suddenly builds on Jenkins started to fail and I figured that happened
>> when parallel builds took place and were wiping out each other's cache.
>> Printing a few debug statements before and after the <ivy:settings> task
>> led me to realize that the cache wasn't being set. That's when I read
>> through the <ivy:settings> task documentation and discovered that
>> <ivy:configure> did more or less the same thing, but that the
>> <ivy:configure> task did the configuration immediately instead of "when
>> needed". Switching to <ivy:configure> fixed the issue.
>>
>>> Also, I'd make sure that ivy:settings is always getting called before
>>> ivy:cleancache.
>>
>> It is. My Ivy project is a subdirectory to the user's project. The
>> developer puts in the lines "<property name="ivy.dir"
>> value="${basedir}/ivy.dir"/> and <import file="${ivy.dir}/ivy.tasks.xml"/>
>> into their build. The "ivy.task.xml" file calls the "<ivy:settings>" task
>> for the user. It runs before any other task is called. The user calls
>> <ivy:cleancache> this way:
>>
>> <property name="ivy.cleancache" value="true"/>
>>
>> <target name="clean">
>> <if>
>> <istrue value="${ivy.cleancache}"/>
>> <then>
>> <ivy:cleancache/>
>> </then>
>> </if>
>> <delete>
>> <fileset dir="${target.dir}"/>
>> </fileset>
>> </target>
>>
>> The user can override the value of ${ivy.cleancache} via a
>> build.properties file, so the cache isn't cleaned every time the developer
>> runs the "clean" target. Thus, <ivy:settings/> is called first before
>> <ivy:cleancache/>.
>>
>>> It appears that your resolution cache is not being defined on a
>>> per-executor basis. If each executor build has its own target directory,
>>> that's OK. But then, why not do that for both your resolution cache and
>>> your repository cache? So you'd specify resolutionCacheDir and
>>> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
>>
>> The resolution cache needs to be defined so that parallel builds don't
>> interfere with each other. There can also be an issue of a developer using
>> multiple working directories of the same project, and this is also an issue
>> with Jenkins too. The resolution cache gets built pretty quickly, so if you
>> delete it and it has to be built from scratch, it takes maybe another
>> second to do. Thus, putting the resolution cache per checkout resolves the
>> issue of multiple builds sharing the same resolution cache since each will
>> have its own. When the developer does a clean, they delete the "target"
>> directory and the resolution cache. No real problem since it can be quickly
>> rebuilt.
>>
>> The Ivy cache is a different issue. If this cache is deleted, all the jars
>> must be redownloaded which takes time. Besides if project A downloads
>> version 1.3 of foo.jar, and project B also needs that jar, project B
>> doesn't have to download it. That's the whole purpose of the Ivy cache.
>> Thus, the resolution cache is placed per project while the Ivy jar cache is
>> shared between different projects.
>>
>>> I forget if Jenkins checks out the source into a different location for
>>> each executor build (it should), but if that's the case and you're
>> sending
>>> your output to a location relative to your source, then establishing a
>>> relative path to your caches should suffice, and there's no need to
>>> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now
>> that I
>>> think of it, specifying Ivy caches for a CI build that are based on
>>> ivy.default.ivy.user.dir, considering that its default is
>> user.home/.ivy2/,
>>> is a bit of an antipattern.
>>
>> No, each job has an established working directory and it uses the same
>> working directory no matter which executor is running. This makes sense
>> since the first time a job might run, it runs on executor #1 and the second
>> time on executor #0. If there were separate working directories per
>> executor, you couldn't take advantage of a version control update which
>> only has to copy a few files over instead of a whole new working directory.
>>
>> The "cache" really doesn't exist on Jenkins because I wipe it out for each
>> build. That's why each executor needs a separate build. I could just as
>> easily put that under the "target" directory like I do the resolution
>> cache. Unfortunately, the same build.xml is also used by the developer.
>> Their machines are slower and on a slower network connection than the
>> servers. Jenkins can download all of the jars in seconds. On the
>> developer's machines, it can take minutes. Thus, the cache is very
>> important to the developers.
>>
>> This is the problem: Both developers and Jenkins use the same build
>> process, and the same build files. But, both use the Ivy cache very
>> differently. Developers want it to act like a cache: Saving them valuable
>> time by not having to redownload files over and over. Jenkins wants a clean
>> slate to make sure that there isn't a jar issue during its build process.
>> Satisfying these two completely different requirements while using the same
>> build process and files lead me to this design.
>>
>> On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mgit...@gmail.com> wrote:
>>> Let me take a stab at addressing your immediate problem, even though I
>>> don't have a direct answer. But then I'm going to suggest a different
>>> approach that avoids that problem.
>>>
>>> Frankly, I've never used configure, only settings. The configure
>>> documentation explains the primary difference (basically settings is
>> lazy):
>>> http://ant.apache.org/ivy/history/latest-milestone/use/configure.html
>>>
>>> As you describe, the immediate problem is that, with settings, the
>>> EXECUTOR_NUMBER environment variable is not being picked up. The
>> following
>>> line is really just establishing a default:
>>> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>>>
>>> For debugging's sake, I'd try leaving that out. The executor number
>> should
>>> be required.
>>>
>>> For debugging's sake, I'd also try outputting the value in the Ant build
>>> just prior to calling ivy:settings.
>>>
>>> Also, I'd make sure that ivy:settings is always getting called before
>>> ivy:cleancache.
>>>
>>> And there's something I find odd about your caches specification:
>>> <caches
>>>
>> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
>>> resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
>>>
>>> It appears that your resolution cache is not being defined on a
>>> per-executor basis. If each executor build has its own target directory,
>>> that's OK. But then, why not do that for both your resolution cache and
>>> your repository cache? So you'd specify resolutionCacheDir and
>>> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
>>>
>>> I forget if Jenkins checks out the source into a different location for
>>> each executor build (it should), but if that's the case and you're
>> sending
>>> your output to a location relative to your source, then establishing a
>>> relative path to your caches should suffice, and there's no need to
>>> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now
>> that I
>>> think of it, specifying Ivy caches for a CI build that are based on
>>> ivy.default.ivy.user.dir, considering that its default is
>> user.home/.ivy2/,
>>> is a bit of an antipattern.
>>>
>>> On Thu, Nov 29, 2012 at 8:35 AM, David Weintraub <qazw...@gmail.com>
>> wrote:
>>>
>>>> When is it better to use <ivy:settings> vs. <ivy:configure>?
>>>>
>>>> I ask because I was having a problem with my Ivy setup. We use
>> Subversion
>>>> and have multiple projects setup that use Ant. In order to centralize
>> our
>>>> jar dependencies, I've decided to implement Ivy into our process.
>>>>
>>>> I did this by creating an Ivy project (
>> https://github.com/qazwart/ivy.dir)
>>>> in Subversion. Users merely set `svn:externals` to include this project,
>>>> make some minor changes in their build.xml, and add in a "ivy.xml" file,
>>>> and everything is set. I control the ivysettings.xml file, so I point
>> Ivy
>>>> to our repository. The Ivy jar is part of the Ivy project, so users
>> don't
>>>> have to install it. The idea was to make Ivy integration as painless as
>>>> possible.
>>>>
>>>> We use Jenkins as our build system, and I decided it would be good if we
>>>> could clean the Ivy cache before each build. However, out Jenkins system
>>>> has six executors, so as many as six builds could happen at once.
>> Cleaning
>>>> the cache in one build while another one runs could cause problems, so I
>>>> modified my ivysettings.xml file to use a different cache depending upon
>>>> the executor:
>>>>
>>>> <ivysettings>
>>>> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>>>> <caches
>>>>
>>>>
>> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
>>>> resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
>>>> <settings defaultResolver="default"/>
>>>> <include file="${ivy.dir}/ivysettings-public.xml"/>
>>>> <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
>>>> <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
>>>> <include
>> url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
>>>> <include
>>>> url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
>>>> </ivysettings>
>>>>
>>>> The problem is that if I did an <ivy:cleancache>, the cache ID was
>> always
>>>> set to "cache-0". When I switched from "<ivy:settings>" to
>>>> "<ivy:configure>", the problem went away. (In the standard build
>> process,
>>>> the <ivy:cleancache> task is executed before the <ivy:resolve> task).
>>>>
>>>> I'm told that "<ivy:settings>" can do multiple configurations, but it
>> looks
>>>> like <ivy:configure> also can do multiple settings too. I also read that
>>>> "<ivy:configure>" was deprecated in some mailings, but it doesn't state
>>>> that in the on line documentation.
>>>>
>>>> So what is the difference between <ivy:settings> and <ivy:configure>,
>> and
>>>> when should I use one over the other?
>>>>
>>>> --
>>>> David Weintraub
>>>> qazw...@gmail.com
>>>>
>>
>>
