RE: cache busting and integration question
Very interesting. After reading this, I configured one cache per repository and
I do believe it fixed one of odd behaviors that I saw earlier.
The odd behavior was an artifact was selected as a winner from the local
repository, but a subsequent last second check for that artifact in the cache
resulted in a cached artifact from a remote repository being delivered. This no
longer seems to be an issue but I am also testing a few other changes so can't
be sure if this is fixed and by this change. But I think it is.
Now I can't imagine why I would ever want a shared cache for multiple
repositories. I think I like one cache per repo much better.
Thanks for the input. It made a tangible positive difference in my testing.
I am currently testing a potential bug fix in the ChainResolver. It might not
be a bug, in which case it would make a great enhancement. Either way, I like
it very much.
The fix is to get the chain resolver to return the latest snapshot from any of
all configured repositories. Some code looked odd in that the intent looked to
be to return the most recent artifact given two artifacts, but no date
comparison was ever performed. Rather, the method always returned the same
Boolean regardless of the age of the artifacts. Performing an actual date
comparison at that point unlocked a bunch of great behaviors.
It was a 4 line fix in ChainResolver. I will share asap.
L.K.
-Original Message-
From: Sven Zethelius [mailto:sv...@expedia.com]
Sent: Monday, March 23, 2015 4:07 PM
To: ivy-user@ant.apache.org
Subject: Re: cache busting and integration question
There are uses where a FileSystemRepository need to be cached for performance.
On Windows, UNC path files will be resolved by the FileSystemRepository. On
Linux, smb or NFS file mounts will benefit from local caching as well.
Have you tried configuring caching differently for each repository in your
ivy.settings.xml. Here¹s a snippet from mine. We make use of the local cache
as a repository when the build is given special instructions to be completely
offline.
caches default=remote-cache
defaultCacheDir=${ivycache.base}
ivyPattern=[organisation]/[module]/ivy-[revision].xml
artifactPattern=[organisation]/[module]/[artifact]-[revision].[ext]
cache name=local-cache useOrigin=true /
cache name=remote-cache useOrigin=false
defaultTTL=${ivycache.timeout} /
cache name=offline-cache
basedir=${ivycache.base}/../offline
useOrigin=true /
/caches
Then in each resolver, I specify which of Œlocal-cache¹, Œremote-cache¹,
Œoffline-cache¹ it should use using the cache attribute.
On 3/23/15, 3:09 PM, Loren Kratzke lkrat...@blueorigin.com wrote:
Ivy works perfectly with a single repository or even a chain of
repositories provided that a dependency does not try to come from two
places.
Ivy records where an artifact comes from but does not take into account
that it may come from a different place next time. When Ivy finds a
candidate artifact in a repository somewhere, it checks the .ivy cache
for that artifact, finds an artifact in the cache (that happens to
originate from somewhere else), doesn't check or realize that it is
from somewhere else, checks the module descriptor against the origin
and determines that the stale artifact is up to date with its stale
origin, and then assumes that the stale artifact is the same as the
found artifact. It does not compare the locations and invalidate the
cached artifact. Rather, it delivers the stale artifact that originated
from a different location than the found artifact.
THAT is a bug. It has to be a bug. There is no way that this is not a
bug. Got that out of my system. Whew!
That said, using a local repo and force tends to overcome this
behavior (in theory, and in my testing, and only sometimes). Results of
whether you get a fresh artifact range from never, to the second
time and after that, to always but only ever from one place
depending upon the configuration.
I think this is fixable or at least patchable though.
For example, when I find an artifact in a file system repository, WHY
would I want to check my cache for that artifact? The artifact is
already right-freaking-there on my file system. It is here. It can't
be any closer without already being delivered to my project. Only
trouble can come from querying the cache when you have the desired
artifact in your hand already. And to then proceed to find and return
an older artifact from the cache that came from a different repo is just not
cool at all.
**Should not check the cache for an artifact that was found in a file
system repository.
**Should not assume that cached artifact came from same repo as found
artifact.
**Should at least check the date to know that the cached artifact about
to be returned is older than the found artifact.
Perhaps using a different
Re: cache busting and integration question
It's an interesting problem to investigate, and I understand your
frustration. But since we don't have exactly the same setup as you, it's
not evident to tell exactly what the problem is and how you can solve it.
We also have an environment with a local and a central repo, but we don't
use the checkmodified attribute. If it would work, it is indeed a good
idea to setup such a resolver chain with SNAPSHOT builds for the developers.
In our case (without checkmodified=true) the detection of changes in both
local and central repo works correctly for us.
Now, I can only search with you in the docs hoping to find a missing
setting (if it's not an actual bug as you suggested).
I suppose that the publication date of the ivy.xml that you published in
your local repo is more recent than the one in your cache ? Did you also
try setting the changing attribute directly in the dependency itself
(allthough I expect the behaviour will be similar).
Could you try a setup where you only publish and resolve from the local
repo (so not chained with your Nexus) ?
You also wrote that latest.integration is a non-starter for you ? So could
you describe how you specify your dependencies ? If you specify an exact
revision in your dependency (like rev=1.0.0-SNAPSHOT), then maybe you
could also try to set the attribute alwaysCheckExactRevision in the
settings of your resolver.
Regards,
Marc
2015-03-23 17:33 GMT+01:00 Loren Kratzke lkrat...@blueorigin.com:
I am not getting the results that you describe. I am testing with a simple
project that publishes a text file with the current date and time so it is
very easy to see when I get a fresh or stale artifact. When I run using the
configuration below, I get the following results when I publish my
dependency and then consume it in the downstream project:
[pub nexus] [get nexus] good
[pub nexus] [get nexus] good
[pub local] [get cached nexus artifact] bad
[pub nexus] [get cached nexus artfiact] bad
So as the first two tests indicate, I can get the latest snapshot from
Nexus. That will work well for the integration server if we leave the local
repo out.
But as the 3rd and 4th tests indicate, if I do a local publish then
everything goes to hell. I do not get my local artifact, I get the cached
artifact. And if I publish to Nexus again, I still get the cached artifact.
In fact I get the cached artifact forever after a local publish no matter
what gets published or where.
That can't be desired behavior. There has to be a bug here somewhere, but
I would be most happy with a work around. This behavior was witnessed when
I debugged. It would find a local artifact, check its cache to find an ivy
file that points at Nexus, and ultimately return a stale Nexus artifact
from the cache.
My setup is very simple - One Nexus repository and one local repository.
Nothing fancy here. I have tried every combination of checkmodified,
changingPattern, and force that the configuration can tolerate without
it quacking a fur ball back at me but I cannot seem to get the desired
behavior. Please find the err in my ways and show me where I have wandered
off the ranch.
I thought I had this working - once - and I changed something and can't
seem to get it back. I thought I had it working as you described where it
would pick up the snapshots, then it would pick up my local artifact, and
it would always deliver the local artifact if present until the local cache
was torched. That behavior is ok with me if I could get it back.
Here is my settings.xml
ivysettings
caches defaultCacheDir=${user.home}/.ivy2/cache/
settings defaultResolver=chain/
credentials host=nexus.mycompany.com realm=Sonatype Nexus
Repository Manager username=name passwd=pass/
property name=m2pattern
value=[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]/
property name=internal.repo.dir value=${user.home}/.ivy2/m2/
property name=mycompany.repo.url value=
http://nexus.mycompany.com:8081/nexus/content/repositories/
resolvers
filesystem name=internal force=true checkmodified=true
changingPattern=.*
ivy
pattern=${internal.repo.dir}/[module]/ivy-[revision].xml /
artifact pattern=${internal.repo.dir}/${m2pattern} /
/filesystem
url name=nexus-snapshots-noplatform m2compatible=true
checkmodified=true changingPattern=.*
artifact
pattern=${mycompany.repo.url}/snapshots/${m2pattern} /
/url
chain name=chain checkmodified=true
resolver ref=internal/
resolver ref=nexus-snapshots-noplatform/
/chain
/resolvers
/ivysettings
L.K.
-Original Message-
From: Zac Jacobson [mailto:pie@gmail.com]
Sent: Sunday, March 22, 2015 6:07 PM
To: ivy-user@ant.apache.org
Subject: Re: cache busting and integration question
If you set your resolver’s checkmodified flag to true, it will verify the
timestamp
RE: cache busting and integration question
Ivy works perfectly with a single repository or even a chain of repositories provided that a dependency does not try to come from two places. Ivy records where an artifact comes from but does not take into account that it may come from a different place next time. When Ivy finds a candidate artifact in a repository somewhere, it checks the .ivy cache for that artifact, finds an artifact in the cache (that happens to originate from somewhere else), doesn't check or realize that it is from somewhere else, checks the module descriptor against the origin and determines that the stale artifact is up to date with its stale origin, and then assumes that the stale artifact is the same as the found artifact. It does not compare the locations and invalidate the cached artifact. Rather, it delivers the stale artifact that originated from a different location than the found artifact. THAT is a bug. It has to be a bug. There is no way that this is not a bug. Got that out of my system. Whew! That said, using a local repo and force tends to overcome this behavior (in theory, and in my testing, and only sometimes). Results of whether you get a fresh artifact range from never, to the second time and after that, to always but only ever from one place depending upon the configuration. I think this is fixable or at least patchable though. For example, when I find an artifact in a file system repository, WHY would I want to check my cache for that artifact? The artifact is already right-freaking-there on my file system. It is here. It can't be any closer without already being delivered to my project. Only trouble can come from querying the cache when you have the desired artifact in your hand already. And to then proceed to find and return an older artifact from the cache that came from a different repo is just not cool at all. **Should not check the cache for an artifact that was found in a file system repository. **Should not assume that cached artifact came from same repo as found artifact. **Should at least check the date to know that the cached artifact about to be returned is older than the found artifact. Perhaps using a different cache directory for each repository would help on that last point, but there is nothing that makes me think that Ivy would not search all of its caches and produce a similar outcome of providing a stale artifact fetched at an earlier time and from a different repository. I may be a bit jaded at this point. But I have many hours in on this so can vent a little. Part of the problem is that Ivy developers somewhere along the line separated the module descriptor from the module that it describes, and both of those from the repository which provided each, recording the origin location of the artifact in the cached module descriptor instead of using independent cache directories per repo and using metadata files, and then not leveraging that origin information to know that it has totally not found a cached version of a found artifact. That's 4 strikes that all add up to a very broken and at times quite silly cache mechanism. I am banking on some of this being a bug and some of it being by design, but mostly bug. When I go back into Ivy source code, I am going to see if I can disable the cache checking when an artifact is found using a file system resolver. That would be a major step forward and would make much of this odd behavior go away. L.K. -Original Message- From: Marc De Boeck [mailto:mdeb...@gmail.com] Sent: Monday, March 23, 2015 1:29 PM To: ivy-user@ant.apache.org Subject: Re: cache busting and integration question It's an interesting problem to investigate, and I understand your frustration. But since we don't have exactly the same setup as you, it's not evident to tell exactly what the problem is and how you can solve it. We also have an environment with a local and a central repo, but we don't use the checkmodified attribute. If it would work, it is indeed a good idea to setup such a resolver chain with SNAPSHOT builds for the developers. In our case (without checkmodified=true) the detection of changes in both local and central repo works correctly for us. Now, I can only search with you in the docs hoping to find a missing setting (if it's not an actual bug as you suggested). I suppose that the publication date of the ivy.xml that you published in your local repo is more recent than the one in your cache ? Did you also try setting the changing attribute directly in the dependency itself (allthough I expect the behaviour will be similar). Could you try a setup where you only publish and resolve from the local repo (so not chained with your Nexus) ? You also wrote that latest.integration is a non-starter for you ? So could you describe how you specify your dependencies ? If you specify an exact revision in your dependency (like rev=1.0.0-SNAPSHOT), then maybe you could also try to set the attribute
RE: cache busting and integration question
://nexus.mycompany.com:8081/nexus/content/repositories/
!--property name=ivy.resolver.default.check.modified value=true/--
resolvers
filesystem name=internal force=true checkmodified=true
changingPattern=.*
ivy pattern=${internal.repo.dir}/[module]/ivy-[revision].xml /
artifact pattern=${internal.repo.dir}/${m2pattern} /
/filesystem
url name=nexus-snapshots-noplatform m2compatible=true
checkmodified=true changingPattern=.*
artifact pattern=${mycompany.repo.url}/snapshots/${m2pattern} /
/url
chain name=chain checkmodified=true changingPattern=.*
resolver ref=internal/
resolver ref=nexus-snapshots-noplatform/
/chain
/resolvers
/ivysettings
*** END ivysettings.xml ***
*** BEGIN ivy.xml for dependency project ***
?xml version=1.0 encoding=utf-8?
ivy-module version=2.0
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:noNamespaceSchemaLocation=http://ant.apache.org/ivy/schemas/ivy.xsd;
xmlns:m=http://ant.apache.org/ivy/maven;
info organisation=example.scratch module=mydep/
publications
artifact name=mydep type=pom/
artifact name=mydep type=txt/
/publications
/ivy-module
*** END ivy.xml for dependency project ***
*** BEGIN ivy.xml for consumer of dependency ***
?xml version=1.0 encoding=utf-8?
ivy-module version=2.0
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:noNamespaceSchemaLocation=http://ant.apache.org/ivy/schemas/ivy.xsd;
xmlns:m=http://ant.apache.org/ivy/maven;
info organisation=example.scratch module=myapp/
publications
artifact name=myapp type=pom/
artifact name=myapp type=txt/
/publications
dependencies
dependency org=example.scratch name=mydep rev=1.0.0-SNAPSHOT
artifact name=mydep type=txt/
/dependency
/dependencies
/ivy-module
*** END ivy.xml for consumer of dependency ***
L.K.
-Original Message-
From: Zac Jacobson [mailto:pie@gmail.com]
Sent: Sunday, March 22, 2015 6:07 PM
To: ivy-user@ant.apache.org
Subject: Re: cache busting and integration question
If you set your resolver’s checkmodified flag to true, it will verify the
timestamp of the published artifact and pull down a more recent version if the
repo is newer than the cache. I’ve found that if you have chained resolvers,
you need to do this at the top level and it affects all repos, not just your
snapshot repository.
Also, if you set the force attribute for your local resolver, it will always
take a version from there rather than going to other resolvers in the chain.
Once you’ve got changes published to your snapshot or release repo, you’ll want
to clean that artifact out of your local repo.
RE: cache busting and integration question
/ resolver ref=nexus-snapshots-noplatform/ /chain /resolvers /ivysettings L.K. -Original Message- From: Zac Jacobson [mailto:pie@gmail.com] Sent: Sunday, March 22, 2015 6:07 PM To: ivy-user@ant.apache.org Subject: Re: cache busting and integration question If you set your resolver’s checkmodified flag to true, it will verify the timestamp of the published artifact and pull down a more recent version if the repo is newer than the cache. I’ve found that if you have chained resolvers, you need to do this at the top level and it affects all repos, not just your snapshot repository. Also, if you set the force attribute for your local resolver, it will always take a version from there rather than going to other resolvers in the chain. Once you’ve got changes published to your snapshot or release repo, you’ll want to clean that artifact out of your local repo. http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html On Mar 20, 2015, at 15:12, Loren Kratzke lkrat...@blueorigin.com wrote: I am encountering challenges with the Ivy cache and also with setting up what I would consider to be a typical (if not classic) developer workflow. Here is the desired workflow: I have a Nexus repository (releases and snapshots) plus a local file system repository used for local development. I have Project which depends upon Library. I wish to modify Library, publish locally, and then pull it into the local build of Project. And for my Jenkins continuous integration server, if the copy of Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the build machine, then I would like Ivy to grab the newer version (just like Maven does). ISSUE #1 Assuming I start with an empty cache and empty internal repo on my local dev machine, I build Project and it pulls Library from Nexus. Perfect. Works great. But then if I publish Library locally and build Project again, I will get the cached version of Library that came from Nexus. No matter what I do, I will always get the cached copy. This happens even though my chain resolver specifies my internal repo before the Nexus repo. If I blow away the cache, then I get my locally published build, but only if I blow away the cache. Otherwise I get stale stuff. Issue #2 below is all about stale stuff. I debugged Ivy (for hours upon hours) and during the resolve, it checks my internal repo, finds the fresh artifact (yay!) and the next thing it does is checks its cache, finds a module descriptor that points at Nexus, and proceeds to pull in the wrong artifact into Project. The behavior is either a bug, or implies that Ivy assumes that an artifact can only come from one place, and if it came from there once, it will come from there forever, and will never change. Again, Ivy finds the artifact in local repo, stops searching, then ends up delivering the cached artifact that came from Nexus sometime earlier. ISSUE #2 I have been working on this problem for quite some time and I thought I had it fixed, but I don't. That is, my integration server needs the latest snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached version from the Ivy cache. An SVN commit happens, triggers a build of Library, Library get published, triggers a build of Project, and a stale build of Library is delivered to Project from the cache (which breaks Project build). SUMMARY Ivy will resolve and cache deps, but once a dep is in the cache, that's what I get forever until I blow away the cache. I see this question asked quite often regarding obtaining latest integration versions, and I see that Ivy claims to support the local development workflow, but I have yet to see a working example of either of these. Surely some of you have solved these issues. Is there any place anywhere in the world that has an example of development workflow configuration and/or integration configuration? Can anybody provide an example? So far I have not found anything, and advice on the net seems to be general suggestions like try this, try that and nobody really has the configuration solution. Is Ivy capable of these behaviors or do I need to hand roll a dependency management solution. Maven is too strict for these projects (C++ code). Ivy is good if I can just obtain these fundamental behaviors. Note that I have not provided any configs in this message that demonstrate the issues I am having but definitely can upon request. Thanks in advance, L.K.
Re: cache busting and integration question
: Zac Jacobson [mailto:pie@gmail.com] Sent: Sunday, March 22, 2015 6:07 PM To: ivy-user@ant.apache.org Subject: Re: cache busting and integration question If you set your resolver’s checkmodified flag to true, it will verify the timestamp of the published artifact and pull down a more recent version if the repo is newer than the cache. I’ve found that if you have chained resolvers, you need to do this at the top level and it affects all repos, not just your snapshot repository. Also, if you set the force attribute for your local resolver, it will always take a version from there rather than going to other resolvers in the chain. Once you’ve got changes published to your snapshot or release repo, you’ll want to clean that artifact out of your local repo. http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html On Mar 20, 2015, at 15:12, Loren Kratzke lkrat...@blueorigin.com wrote: I am encountering challenges with the Ivy cache and also with setting up what I would consider to be a typical (if not classic) developer workflow. Here is the desired workflow: I have a Nexus repository (releases and snapshots) plus a local file system repository used for local development. I have Project which depends upon Library. I wish to modify Library, publish locally, and then pull it into the local build of Project. And for my Jenkins continuous integration server, if the copy of Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the build machine, then I would like Ivy to grab the newer version (just like Maven does). ISSUE #1 Assuming I start with an empty cache and empty internal repo on my local dev machine, I build Project and it pulls Library from Nexus. Perfect. Works great. But then if I publish Library locally and build Project again, I will get the cached version of Library that came from Nexus. No matter what I do, I will always get the cached copy. This happens even though my chain resolver specifies my internal repo before the Nexus repo. If I blow away the cache, then I get my locally published build, but only if I blow away the cache. Otherwise I get stale stuff. Issue #2 below is all about stale stuff. I debugged Ivy (for hours upon hours) and during the resolve, it checks my internal repo, finds the fresh artifact (yay!) and the next thing it does is checks its cache, finds a module descriptor that points at Nexus, and proceeds to pull in the wrong artifact into Project. The behavior is either a bug, or implies that Ivy assumes that an artifact can only come from one place, and if it came from there once, it will come from there forever, and will never change. Again, Ivy finds the artifact in local repo, stops searching, then ends up delivering the cached artifact that came from Nexus sometime earlier. ISSUE #2 I have been working on this problem for quite some time and I thought I had it fixed, but I don't. That is, my integration server needs the latest snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached version from the Ivy cache. An SVN commit happens, triggers a build of Library, Library get published, triggers a build of Project, and a stale build of Library is delivered to Project from the cache (which breaks Project build). SUMMARY Ivy will resolve and cache deps, but once a dep is in the cache, that's what I get forever until I blow away the cache. I see this question asked quite often regarding obtaining latest integration versions, and I see that Ivy claims to support the local development workflow, but I have yet to see a working example of either of these. Surely some of you have solved these issues. Is there any place anywhere in the world that has an example of development workflow configuration and/or integration configuration? Can anybody provide an example? So far I have not found anything, and advice on the net seems to be general suggestions like try this, try that and nobody really has the configuration solution. Is Ivy capable of these behaviors or do I need to hand roll a dependency management solution. Maven is too strict for these projects (C++ code). Ivy is good if I can just obtain these fundamental behaviors. Note that I have not provided any configs in this message that demonstrate the issues I am having but definitely can upon request. Thanks in advance, L.K.
RE: cache busting and integration question
I am not getting the results that you describe. I am testing with a simple
project that publishes a text file with the current date and time so it is very
easy to see when I get a fresh or stale artifact. When I run using the
configuration below, I get the following results when I publish my dependency
and then consume it in the downstream project:
[pub nexus] [get nexus] good
[pub nexus] [get nexus] good
[pub local] [get cached nexus artifact] bad
[pub nexus] [get cached nexus artfiact] bad
So as the first two tests indicate, I can get the latest snapshot from Nexus.
That will work well for the integration server if we leave the local repo out.
But as the 3rd and 4th tests indicate, if I do a local publish then everything
goes to hell. I do not get my local artifact, I get the cached artifact. And if
I publish to Nexus again, I still get the cached artifact. In fact I get the
cached artifact forever after a local publish no matter what gets published or
where.
That can't be desired behavior. There has to be a bug here somewhere, but I
would be most happy with a work around. This behavior was witnessed when I
debugged. It would find a local artifact, check its cache to find an ivy file
that points at Nexus, and ultimately return a stale Nexus artifact from the
cache.
My setup is very simple - One Nexus repository and one local repository.
Nothing fancy here. I have tried every combination of checkmodified,
changingPattern, and force that the configuration can tolerate without it
quacking a fur ball back at me but I cannot seem to get the desired behavior.
Please find the err in my ways and show me where I have wandered off the ranch.
I thought I had this working - once - and I changed something and can't seem to
get it back. I thought I had it working as you described where it would pick up
the snapshots, then it would pick up my local artifact, and it would always
deliver the local artifact if present until the local cache was torched. That
behavior is ok with me if I could get it back.
Here is my settings.xml
ivysettings
caches defaultCacheDir=${user.home}/.ivy2/cache/
settings defaultResolver=chain/
credentials host=nexus.mycompany.com realm=Sonatype Nexus Repository
Manager username=name passwd=pass/
property name=m2pattern
value=[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]/
property name=internal.repo.dir value=${user.home}/.ivy2/m2/
property name=mycompany.repo.url
value=http://nexus.mycompany.com:8081/nexus/content/repositories/
resolvers
filesystem name=internal force=true checkmodified=true
changingPattern=.*
ivy pattern=${internal.repo.dir}/[module]/ivy-[revision].xml /
artifact pattern=${internal.repo.dir}/${m2pattern} /
/filesystem
url name=nexus-snapshots-noplatform m2compatible=true
checkmodified=true changingPattern=.*
artifact pattern=${mycompany.repo.url}/snapshots/${m2pattern} /
/url
chain name=chain checkmodified=true
resolver ref=internal/
resolver ref=nexus-snapshots-noplatform/
/chain
/resolvers
/ivysettings
L.K.
-Original Message-
From: Zac Jacobson [mailto:pie@gmail.com]
Sent: Sunday, March 22, 2015 6:07 PM
To: ivy-user@ant.apache.org
Subject: Re: cache busting and integration question
If you set your resolver’s checkmodified flag to true, it will verify the
timestamp of the published artifact and pull down a more recent version if the
repo is newer than the cache. I’ve found that if you have chained resolvers,
you need to do this at the top level and it affects all repos, not just your
snapshot repository.
Also, if you set the force attribute for your local resolver, it will always
take a version from there rather than going to other resolvers in the chain.
Once you’ve got changes published to your snapshot or release repo, you’ll want
to clean that artifact out of your local repo.
http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html
http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html
On Mar 20, 2015, at 15:12, Loren Kratzke lkrat...@blueorigin.com wrote:
I am encountering challenges with the Ivy cache and also with setting up what
I would consider to be a typical (if not classic) developer workflow. Here is
the desired workflow:
I have a Nexus repository (releases and snapshots) plus a local file system
repository used for local development. I have Project which depends upon
Library. I wish to modify Library, publish locally, and then pull it into the
local build of Project.
And for my Jenkins continuous integration server, if the copy of
Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the
build machine, then I would like Ivy to grab the newer version (just like
Maven does).
ISSUE #1
Assuming I start with an empty cache and empty
RE: cache busting and integration question
I actually do implement a dual publish. I publish a unique snapshot that includes an svn rev number in the version number, and then a generic snapshot that does not. I publish the same artifact twice. For example, myartifact-1.0.0_5353-SNAPSHOT and myartifact-1.0.0-SNAPSHOT. If you want the latest, depend upon the latter (sans the cache issues that I am having). The latter is always the latest build and not a specific build. That is by design. I want to be able to perform an integration build against the most recent commits (every commit if possible) and break early if there is a compilation problem. Jenkins will report the latest commit that broke the build. I don't need a unique version number to figure that part out. Then for functional testing, we use the uniquely versioned artifacts for obvious reasons. You can't test a moving target. What I wish to avoid is having Ivy evaluate version numbers and determine which is the latest version for integration purposes. Developers often like to get their freak on when it comes to version numbering and eventually Ivy is not going to like it. The only way to determine the latest version of something with certainty is in the time domain. Latest means most recent in time not alphabetically last, or numerically larger, or it matches my funky regex, or anything else. It is nice that Ivy can attempt to determine the latest version, and perhaps it does so very well, but eventually actual lives will be in the balance and blaming a developer for creating a breaking version number will not be an option for me. I have no choice but to use clock time to determine what the latest version of something is. So latest.integration is a non-starter for me. Please do not take that as a lecture. I am just explaining my constraints in painful detail. I have been over the pages that you referenced many times. And again, this response is like so many other Ivy responses. It points to a page, which points at configuration documentation, which fails to provide a complete example of the desired behavior. Within the docs, it tells me what you said, that I should really use unique version numbers. And above I have explained in great detail that I do, and why I cannot allow Ivy to interpret those numbers to determine the latest version. I can only use actual time. Nothing less. In my other response, I displayed a configuration that correctly obtains the latest snapshot from Nexus every time until a local publish to the internal repo happens. Then Ivy delivers only the cached version from the external repo forever (not even the local publish). It will no longer pick up new snapshots from the Nexus repo, and will not deliver the locally published artifact. When debugging Ivy, Ivy would find the artifact in my internal repository and proceed to deliver the stale artifact from the cache thinking that it is a cached version of the artifact from the internal repository, but it is actually a cached version of the Nexus artifact. I am fairly certain that Ivy is broken. This is definitely a bug. Either that, or Ivy assumes that an artifact can only come from one place, and once it comes from that place, it can come from no other place, ever. I don't believe that was the intent of the designers. One last possibility is that I have conjured up a very unfortunate combination of configuration attributes. I hope that this is the case. I appreciate the pointers to the docs. I am sorry but they fall short otherwise I would have had this working by now. I need a working config, not more Ivy docs. I have been reading those for a year now. Yes, it is possible that I am missing something. I hope that I am. But all indications are that Ivy is broken. Failing a working config, I will move the discussion to the developers list. If Ivy would not substitute the cached version of the incorrect artifact for the found artifact at the last moment, everything would work properly. L.K. -Original Message- From: Marc De Boeck [mailto:mdeb...@gmail.com] Sent: Sunday, March 22, 2015 2:30 PM To: ivy-user@ant.apache.org Subject: Re: cache busting and integration question Please have a look at the paragraphs Dealing with integration versions in http://ant.apache.org/ivy/history/latest-milestone/bestpractices.html: The typical developer workflow using SNAPSHOT builds of a release, has become popular because it is the standard in Maven. A disadvantage of using SNAPSHOT-builds, is however that you can never be sure which build you have finally gotten after a dependency has been resolved. A more controlled way of dealing with your dependencies, is to create a new version for every build that is made, and to resolve for your local builds using the revision 'latest.integration' for home-made artifacts. Ivy supports both methods of resolving dependencies. The URL http://ant.apache.org/ivy/history/latest-milestone/bestpractices.html describes how you can
Re: cache busting and integration question
There are uses where a FileSystemRepository need to be cached for
performance. On Windows, UNC path files will be resolved by the
FileSystemRepository. On Linux, smb or NFS file mounts will benefit from
local caching as well.
Have you tried configuring caching differently for each repository in your
ivy.settings.xml. Here¹s a snippet from mine. We make use of the local
cache as a repository when the build is given special instructions to be
completely offline.
caches default=remote-cache
defaultCacheDir=${ivycache.base}
ivyPattern=[organisation]/[module]/ivy-[revision].xml
artifactPattern=[organisation]/[module]/[artifact]-[revision].[ext]
cache name=local-cache useOrigin=true /
cache name=remote-cache useOrigin=false
defaultTTL=${ivycache.timeout} /
cache name=offline-cache
basedir=${ivycache.base}/../offline
useOrigin=true /
/caches
Then in each resolver, I specify which of Œlocal-cache¹, Œremote-cache¹,
Œoffline-cache¹ it should use using the cache attribute.
On 3/23/15, 3:09 PM, Loren Kratzke lkrat...@blueorigin.com wrote:
Ivy works perfectly with a single repository or even a chain of
repositories provided that a dependency does not try to come from two
places.
Ivy records where an artifact comes from but does not take into account
that it may come from a different place next time. When Ivy finds a
candidate artifact in a repository somewhere, it checks the .ivy cache
for that artifact, finds an artifact in the cache (that happens to
originate from somewhere else), doesn't check or realize that it is from
somewhere else, checks the module descriptor against the origin and
determines that the stale artifact is up to date with its stale origin,
and then assumes that the stale artifact is the same as the found
artifact. It does not compare the locations and invalidate the cached
artifact. Rather, it delivers the stale artifact that originated from a
different location than the found artifact.
THAT is a bug. It has to be a bug. There is no way that this is not a
bug. Got that out of my system. Whew!
That said, using a local repo and force tends to overcome this behavior
(in theory, and in my testing, and only sometimes). Results of whether
you get a fresh artifact range from never, to the second time and
after that, to always but only ever from one place depending upon the
configuration.
I think this is fixable or at least patchable though.
For example, when I find an artifact in a file system repository, WHY
would I want to check my cache for that artifact? The artifact is already
right-freaking-there on my file system. It is here. It can't be any
closer without already being delivered to my project. Only trouble can
come from querying the cache when you have the desired artifact in your
hand already. And to then proceed to find and return an older artifact
from the cache that came from a different repo is just not cool at all.
**Should not check the cache for an artifact that was found in a file
system repository.
**Should not assume that cached artifact came from same repo as found
artifact.
**Should at least check the date to know that the cached artifact about
to be returned is older than the found artifact.
Perhaps using a different cache directory for each repository would help
on that last point, but there is nothing that makes me think that Ivy
would not search all of its caches and produce a similar outcome of
providing a stale artifact fetched at an earlier time and from a
different repository. I may be a bit jaded at this point. But I have many
hours in on this so can vent a little.
Part of the problem is that Ivy developers somewhere along the line
separated the module descriptor from the module that it describes, and
both of those from the repository which provided each, recording the
origin location of the artifact in the cached module descriptor instead
of using independent cache directories per repo and using metadata files,
and then not leveraging that origin information to know that it has
totally not found a cached version of a found artifact. That's 4 strikes
that all add up to a very broken and at times quite silly cache
mechanism.
I am banking on some of this being a bug and some of it being by design,
but mostly bug. When I go back into Ivy source code, I am going to see if
I can disable the cache checking when an artifact is found using a file
system resolver. That would be a major step forward and would make much
of this odd behavior go away.
L.K.
-Original Message-
From: Marc De Boeck [mailto:mdeb...@gmail.com]
Sent: Monday, March 23, 2015 1:29 PM
To: ivy-user@ant.apache.org
Subject: Re: cache busting and integration question
It's an interesting problem to investigate, and I understand your
frustration. But since we don't have exactly the same setup as you, it's
not evident to tell exactly what the problem
RE: cache busting and integration question
That is excellent insight. Thank you for that. The things you found encourage me to fix the issues that I am seeing (or patch them). We are publishing and consuming C++ artifacts using Ivy/Nexus. We also have a Maven side of the house but they use a different Nexus repo dedicated to pure Java artifacts. So I have not seen any Ivy/Maven cross pollination issues. Based upon your comments however I will inspect the metadata files and check into the timestamps. Hopefully the parts that we are using on the Ivy side are ok. I agree about the cache implementation being tied to the resolvers. I considered writing a new resolver but the resolution and download of the artifacts works just fine. It is higher up where the problems begin, like the chain resolver and cache as you mentioned. The one thing that is working for us right now is torching the ivy cache before every build. That does not imply very nice things about the ivy cache functionality. It is more of a liability than as asset to us. I will try to fix this. Ivy is a good technology. It just needs some tough love. L.K. -Original Message- From: Josh Suereth [mailto:joshua.suer...@gmail.com] Sent: Monday, March 23, 2015 2:16 PM To: ivy-user@ant.apache.org Subject: RE: cache busting and integration question We experienced this bug in sbt as well. If I recall correctly, it turns out that the implementation of most resolvers is rather tied to the implementation of the cache. One artifact of this is that the chain resolver wasn't ever looking for -SNAPSHOTs when we wanted it to. We actually did the following, recently - 1. A new chain resolver implementation which forces a date check against the remote repository. It turns out this exposed issues in publication date not being a universal thing. E.g. pom.xml files weren't reporting legitimate publication dates, leading to any attempt to pull latest for a snapshot being wrong. 2. We created a new implementation of DependencyResolver which actually uses aether. This had two side effects: 2a. We avoid parent pom issues which still sneak in when using -SNAPSHOT parents or weird profiles in POM's (why hadoop, why!!!) 2b. We always correctly update maven-metadata.xml and maven-metadata-local.xml. this means we'll have a somewhat legitimate publication date to do loose comparisons on (not exact but good enough). Both are still in experimental stage, but both now pass our previously failing test cases around -SNAPSHOT updates, when used together. What does this mean for ivy? 1. Ivy should probably support maven-metadata(-local).xml directly in its maven code. 2. Ivy chain resolver / ivy cache may need tweaks for dynamic revisions to make sure, even if the version number is the same, that remote repositories (or just metadata.xml) is checked. This adventure lead me to believe that the ivy cache api itself may need a 3.0 overhaul, and the maven code may need to be further separated from core usage. While the loss of reuse would be sad, I think the reality is maven just doesn't work the way ivy was designed. On Mar 23, 2015 12:33 PM, Loren Kratzke lkrat...@blueorigin.com wrote: I am not getting the results that you describe. I am testing with a simple project that publishes a text file with the current date and time so it is very easy to see when I get a fresh or stale artifact. When I run using the configuration below, I get the following results when I publish my dependency and then consume it in the downstream project: [pub nexus] [get nexus] good [pub nexus] [get nexus] good [pub local] [get cached nexus artifact] bad [pub nexus] [get cached nexus artfiact] bad So as the first two tests indicate, I can get the latest snapshot from Nexus. That will work well for the integration server if we leave the local repo out. But as the 3rd and 4th tests indicate, if I do a local publish then everything goes to hell. I do not get my local artifact, I get the cached artifact. And if I publish to Nexus again, I still get the cached artifact. In fact I get the cached artifact forever after a local publish no matter what gets published or where. That can't be desired behavior. There has to be a bug here somewhere, but I would be most happy with a work around. This behavior was witnessed when I debugged. It would find a local artifact, check its cache to find an ivy file that points at Nexus, and ultimately return a stale Nexus artifact from the cache. My setup is very simple - One Nexus repository and one local repository. Nothing fancy here. I have tried every combination of checkmodified, changingPattern, and force that the configuration can tolerate without it quacking a fur ball back at me but I cannot seem to get the desired behavior. Please find the err in my ways and show me where I have wandered off the ranch. I thought I had this working - once - and I changed something and can't seem to get it back. I
Re: cache busting and integration question
Please have a look at the paragraphs Dealing with integration versions in http://ant.apache.org/ivy/history/latest-milestone/bestpractices.html: The typical developer workflow using SNAPSHOT builds of a release, has become popular because it is the standard in Maven. A disadvantage of using SNAPSHOT-builds, is however that you can never be sure which build you have finally gotten after a dependency has been resolved. A more controlled way of dealing with your dependencies, is to create a new version for every build that is made, and to resolve for your local builds using the revision 'latest.integration' for home-made artifacts. Ivy supports both methods of resolving dependencies. The URL http://ant.apache.org/ivy/history/latest-milestone/bestpractices.html describes how you can implement the SNAPSHOT approach using Ivy. Regards, Marc 2015-03-20 23:12 GMT+01:00 Loren Kratzke lkrat...@blueorigin.com: I am encountering challenges with the Ivy cache and also with setting up what I would consider to be a typical (if not classic) developer workflow. Here is the desired workflow: I have a Nexus repository (releases and snapshots) plus a local file system repository used for local development. I have Project which depends upon Library. I wish to modify Library, publish locally, and then pull it into the local build of Project. And for my Jenkins continuous integration server, if the copy of Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the build machine, then I would like Ivy to grab the newer version (just like Maven does). ISSUE #1 Assuming I start with an empty cache and empty internal repo on my local dev machine, I build Project and it pulls Library from Nexus. Perfect. Works great. But then if I publish Library locally and build Project again, I will get the cached version of Library that came from Nexus. No matter what I do, I will always get the cached copy. This happens even though my chain resolver specifies my internal repo before the Nexus repo. If I blow away the cache, then I get my locally published build, but only if I blow away the cache. Otherwise I get stale stuff. Issue #2 below is all about stale stuff. I debugged Ivy (for hours upon hours) and during the resolve, it checks my internal repo, finds the fresh artifact (yay!) and the next thing it does is checks its cache, finds a module descriptor that points at Nexus, and proceeds to pull in the wrong artifact into Project. The behavior is either a bug, or implies that Ivy assumes that an artifact can only come from one place, and if it came from there once, it will come from there forever, and will never change. Again, Ivy finds the artifact in local repo, stops searching, then ends up delivering the cached artifact that came from Nexus sometime earlier. ISSUE #2 I have been working on this problem for quite some time and I thought I had it fixed, but I don't. That is, my integration server needs the latest snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached version from the Ivy cache. An SVN commit happens, triggers a build of Library, Library get published, triggers a build of Project, and a stale build of Library is delivered to Project from the cache (which breaks Project build). SUMMARY Ivy will resolve and cache deps, but once a dep is in the cache, that's what I get forever until I blow away the cache. I see this question asked quite often regarding obtaining latest integration versions, and I see that Ivy claims to support the local development workflow, but I have yet to see a working example of either of these. Surely some of you have solved these issues. Is there any place anywhere in the world that has an example of development workflow configuration and/or integration configuration? Can anybody provide an example? So far I have not found anything, and advice on the net seems to be general suggestions like try this, try that and nobody really has the configuration solution. Is Ivy capable of these behaviors or do I need to hand roll a dependency management solution. Maven is too strict for these projects (C++ code). Ivy is good if I can just obtain these fundamental behaviors. Note that I have not provided any configs in this message that demonstrate the issues I am having but definitely can upon request. Thanks in advance, L.K.
Re: cache busting and integration question
If you set your resolver’s checkmodified flag to true, it will verify the timestamp of the published artifact and pull down a more recent version if the repo is newer than the cache. I’ve found that if you have chained resolvers, you need to do this at the top level and it affects all repos, not just your snapshot repository. Also, if you set the force attribute for your local resolver, it will always take a version from there rather than going to other resolvers in the chain. Once you’ve got changes published to your snapshot or release repo, you’ll want to clean that artifact out of your local repo. http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html On Mar 20, 2015, at 15:12, Loren Kratzke lkrat...@blueorigin.com wrote: I am encountering challenges with the Ivy cache and also with setting up what I would consider to be a typical (if not classic) developer workflow. Here is the desired workflow: I have a Nexus repository (releases and snapshots) plus a local file system repository used for local development. I have Project which depends upon Library. I wish to modify Library, publish locally, and then pull it into the local build of Project. And for my Jenkins continuous integration server, if the copy of Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the build machine, then I would like Ivy to grab the newer version (just like Maven does). ISSUE #1 Assuming I start with an empty cache and empty internal repo on my local dev machine, I build Project and it pulls Library from Nexus. Perfect. Works great. But then if I publish Library locally and build Project again, I will get the cached version of Library that came from Nexus. No matter what I do, I will always get the cached copy. This happens even though my chain resolver specifies my internal repo before the Nexus repo. If I blow away the cache, then I get my locally published build, but only if I blow away the cache. Otherwise I get stale stuff. Issue #2 below is all about stale stuff. I debugged Ivy (for hours upon hours) and during the resolve, it checks my internal repo, finds the fresh artifact (yay!) and the next thing it does is checks its cache, finds a module descriptor that points at Nexus, and proceeds to pull in the wrong artifact into Project. The behavior is either a bug, or implies that Ivy assumes that an artifact can only come from one place, and if it came from there once, it will come from there forever, and will never change. Again, Ivy finds the artifact in local repo, stops searching, then ends up delivering the cached artifact that came from Nexus sometime earlier. ISSUE #2 I have been working on this problem for quite some time and I thought I had it fixed, but I don't. That is, my integration server needs the latest snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached version from the Ivy cache. An SVN commit happens, triggers a build of Library, Library get published, triggers a build of Project, and a stale build of Library is delivered to Project from the cache (which breaks Project build). SUMMARY Ivy will resolve and cache deps, but once a dep is in the cache, that's what I get forever until I blow away the cache. I see this question asked quite often regarding obtaining latest integration versions, and I see that Ivy claims to support the local development workflow, but I have yet to see a working example of either of these. Surely some of you have solved these issues. Is there any place anywhere in the world that has an example of development workflow configuration and/or integration configuration? Can anybody provide an example? So far I have not found anything, and advice on the net seems to be general suggestions like try this, try that and nobody really has the configuration solution. Is Ivy capable of these behaviors or do I need to hand roll a dependency management solution. Maven is too strict for these projects (C++ code). Ivy is good if I can just obtain these fundamental behaviors. Note that I have not provided any configs in this message that demonstrate the issues I am having but definitely can upon request. Thanks in advance, L.K.
