Re: Removing interim dated builds from /www/www.apache.org/dist/java-repository
Dain Sundstrom wrote: On Jan 7, 2005, at 6:56 AM, Mark R. Diggory wrote: Heres a really good example of a reference we promote usage of: jakarta-commons/chain/project.properties,v: maven.repo.central.directory=/www/www.apache.org/dist/java-repository maven.repo.central.directory is property which states "where" to publish to, not where to download from. This is exactly my point. If someone pulled down that source, it potentially won't build. Once you start messing with repos, it can eliminate the possibility of recreating *historical* builds. No, your missing it, the point of "maven.repo.central.directory" is that it specifically configures where things get "published" to, not where to download them from. Release Managers using Maven need to maintain this sort of stuff outside of the cvs and only use it when publishing releases or interim builds http://maven.apache.org/reference/user-guide.html#Behavioural_Properties http://maven.apache.org/reference/plugins/artifact/properties.html Are examples of configuring the client. It might be good to go through the Apache projects that are in java-repository and see if any are packaged to download dependencies from there, I doubt it that here would be any occurances however. As a majority of the files were originally published to/against ibiblio in the first place. Unfortunately, not all projects that use the apache repos are not hosted at apache. We've never publicized the the java-repository to be used externally. It 's content was originally migrated from Apache by Jason Van Zyl and Myself with the goal of getting publishing of Apache content back to Apache (with Ibibilio as a mirror). Anyone using it would be using it at their own risk. I believe that any consultation with the ASF Repository project here at Apache would have made this issue clear to them. Further, my point is that currently there shouldn't be projects dependent on java-repository as its location for resolving jar dependencies, there is an entire discussion on the [EMAIL PROTECTED] list concerning this subject, the repository used is a configuration of the client right now (in Maven) not a characteristic of the project.xml of a maven project. You can suggest a repository to use in properties handed to Maven, but the default is always www.ibiblio.org/maven. Our initial creation of java-repository was never not to have a "Standalone" Maven repository at Apache, but to provide a simple means for Apache developers to publish release jars and distributions to Ibibilio until such a time when distributed repositories become a greater practice. Its a real bad idea to force build dependencies against it at this time until we get it stabilized. Its entire contents are present at Ibiblio, there currently is no reason a project should be using it directly vs ibibilio. I agree with what you are writing, but people do it. IMHO if you are going to create a public repo, they you have a responsibility to keep the artifacts around forever. But this is exactly our argument, java-repository was never meant to be used as a public interim release repository (based on where it is located in the "dist" directory and the policies of "dist"... On top of this I recommend that Apache avoid maintaining interim builds in such a way that external projects can build dependencies against them. External projects should only be using fully 'sanctioned' releases published in dist for this purpose. Until tooling is completed in Ant and Maven to support distributing download requests from the ASF Repository across the mirrors we really cannot recommend that external projects build dependencies against the contents of the ASF Repository directly, please use Ibiblio/maven for this until such capabilities come into existence. That is simply not possible. Geronimo has three external sister projects OpenEJB, ActiveMQ and TranQL. These projects import Geronimo and Geronimo aggregates their jars into the final distribution. If this were policy, then either the Geronimo project would stop, or more likely the Geronimo project would publish fully "sanctioned' releases every checkin. This is an interesting example that represents a challenge. This is probably best served using http://cvs.apache.org/repository or a repository for Geronimo to manage sharing of the development snapshots. Why are they so separate? Why not have these projects as part of Apache? After writing some scripts to search the Apache cvs tree and inspecting the results. It really breaks down to three cases where projects reference java-repository: 1.) In Maven project.properties, Referencing the remote repository maven.repo.remote=http://www.apache.org/dist/java-repository/ 2.) In Maven project.properties, Referencing a central repository maven.repo.central.directory=/www/www.apache.org/dist/java-repository 3.) In Ant files generated from Maven, Hardcoded references to java-repository. I believe we
Re: Removing interim dated builds from /www/www.apache.org/dist/java-repository
On Jan 7, 2005, at 6:56 AM, Mark R. Diggory wrote: Dain Sundstrom wrote: I would hope so. These projects have a trivial or non existent set of dependencies. Take a look at a big project using maven and you will see that they use lots of repos, because ibiblio tends not to have everything necessary or was down when someone was working. Yet, ibiblio is an exact mirror of dist/java-repository without deletion. There is nothing in dist/java-repository that is not also present on ibibilio and the rsync is every four hours so there is actually only a tiny lag concerning differences. Ibiblio is the canonical repository of Maven, you would have to be doing a tremendous about of overriding to change its location on a consistent basis. "Why?" waist such time. Regardless of what people should and shouldn't do, they do this all the time. My understanding is that Ibibilio is a system dedicated to mirroring massive amounts of content with raid/fallover and multiple gateways/ISP's, I would expect its up time to be far more consistent than even that of Apache. :) I would expect that also, but it simply isn't true. Ibiblio has blackouts and brownouts all the time. The latter are much worse as it doesn't immediately return an error code; instead, it spends like 5 minutes downloading a 20k jar. In Maven, the repository used to get downloads from is not a project attiribute, it is a user decision/configuaration of the client. So really, a project that ends up with any dependency downloads directly from dist/java-repository is not the best solution, ibiblio is the more persistent and canonical point to be downloading from at this point. Take a look at geronimo and you will see that it is impractical for a user to hunt down dependencies. The code from a source distribution should just build, which means that all the dependencies *must* always be available. A simple search of the files in the filesystem for any string matches to "dist/java-repository" would suffice. I checked it out and searched it on my laptop just now using Eclipse, there are no references in any of the files in any of geronimo. I been doing greps on cvs.apache.org and find the following cases as examples of references jakarta-turbine-2/project.properties,v:maven.repo.remote=http:// www.ibiblio.org/maven/,http://www.apache.org/dist/java-repository/ This is a bad usage of java-repository, but not a show stopper, It will go to ibilio first then the ASF Repository, things should work fine as long as ibibilio's up. We should make sure there are no cases of the following for example: jakarta-turbine-2/project.properties,v:maven.repo.remote=http:// www.apache.org/dist/java-repository/ Thats bad, theres really no reason this should be done at this point and time. Heres a really good example of a reference we promote usage of: jakarta-commons/chain/project.properties,v: maven.repo.central.directory=/www/www.apache.org/dist/java-repository maven.repo.central.directory is property which states "where" to publish to, not where to download from. This is exactly my point. If someone pulled down that source, it potentially won't build. Once you start messing with repos, it can eliminate the possibility of recreating *historical* builds. It might be good to go through the Apache projects that are in java-repository and see if any are packaged to download dependencies from there, I doubt it that here would be any occurances however. As a majority of the files were originally published to/against ibiblio in the first place. Unfortunately, not all projects that use the apache repos are not hosted at apache. We've never publicized the the java-repository to be used externally. It 's content was originally migrated from Apache by Jason Van Zyl and Myself with the goal of getting publishing of Apache content back to Apache (with Ibibilio as a mirror). Anyone using it would be using it at their own risk. I believe that any consultation with the ASF Repository project here at Apache would have made this issue clear to them. Further, my point is that currently there shouldn't be projects dependent on java-repository as its location for resolving jar dependencies, there is an entire discussion on the [EMAIL PROTECTED] list concerning this subject, the repository used is a configuration of the client right now (in Maven) not a characteristic of the project.xml of a maven project. You can suggest a repository to use in properties handed to Maven, but the default is always www.ibiblio.org/maven. Our initial creation of java-repository was never not to have a "Standalone" Maven repository at Apache, but to provide a simple means for Apache developers to publish release jars and distributions to Ibibilio until such a time when distributed repositories become a greater practice. Its a real bad idea to force build dependencies against it at this time until we get it stabilized. Its entire contents
Re: Removing interim dated builds from /www/www.apache.org/dist/java-repository
Thanks Brett, I'm just concerned as well, that there may be distributions out on Apache the reference the ASF Repository jars directly instead of using Ibiblio or dyncloser.cgi. If those projects have been shortsighted in releasing distros's that have dependencies on interim releases in java-repository, then my proposed changes will effect them. I think I will be moving forward with this proposal. As well as deleting these files, I will build a log of all the changes so that recovery from the archives can occur if any problems arise. It seems more important that we get the repository properly organized and structured than maintain old external project dependencies on unsanctioned interim releases in java-repository. -Mark Brett Porter wrote: Hi Mark, I think this is just a miscommunication, as what you have said below is what all the projects do 1.) In Maven project.properties, Referencing the remote repository maven.repo.remote=http://www.apache.org/dist/java-repository/ 2.) In Maven project.properties, Referencing a central repository maven.repo.central.directory=/www/www.apache.org/dist/java-repository 3.) In Ant files generated from Maven, Hardcoded references to java-repository. I believe we should take the following stance on these cases: 1.) 1 !!! 2.) +1 3.) 1 !!! I agree. As well, most projects that have java-repository as a remote repo, also have have ibibilio in from of it: If I ever reference java-repository, its not in an automated fashion, and always behind dyncloser.cgi (eg, the release announcements). - Brett - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Mark Diggory Software Developer Harvard MIT Data Center http://www.hmdc.harvard.edu - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Removing interim dated builds from /www/www.apache.org/dist/java-repository
Hi Mark, I think this is just a miscommunication, as what you have said below is what all the projects do > > 1.) In Maven project.properties, Referencing the remote repository > > maven.repo.remote=http://www.apache.org/dist/java-repository/ > > 2.) In Maven project.properties, Referencing a central repository > > maven.repo.central.directory=/www/www.apache.org/dist/java-repository > > 3.) In Ant files generated from Maven, Hardcoded references to > java-repository. > > I believe we should take the following stance on these cases: > > 1.) 1 !!! > > 2.) +1 > > 3.) 1 !!! I agree. > As well, most projects that have java-repository as a remote repo, also > have have ibibilio in from of it: If I ever reference java-repository, its not in an automated fashion, and always behind dyncloser.cgi (eg, the release announcements). - Brett - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Removing interim dated builds from /www/www.apache.org/dist/java-repository
Dain Sundstrom wrote: I would hope so. These projects have a trivial or non existent set of dependencies. Take a look at a big project using maven and you will see that they use lots of repos, because ibiblio tends not to have everything necessary or was down when someone was working. Yet, ibiblio is an exact mirror of dist/java-repository without deletion. There is nothing in dist/java-repository that is not also present on ibibilio and the rsync is every four hours so there is actually only a tiny lag concerning differences. Ibiblio is the canonical repository of Maven, you would have to be doing a tremendous about of overriding to change its location on a consistent basis. "Why?" waist such time. My understanding is that Ibibilio is a system dedicated to mirroring massive amounts of content with raid/fallover and multiple gateways/ISP's, I would expect its up time to be far more consistent than even that of Apache. In Maven, the repository used to get downloads from is not a project attiribute, it is a user decision/configuaration of the client. So really, a project that ends up with any dependency downloads directly from dist/java-repository is not the best solution, ibiblio is the more persistent and canonical point to be downloading from at this point. Take a look at geronimo and you will see that it is impractical for a user to hunt down dependencies. The code from a source distribution should just build, which means that all the dependencies *must* always be available. A simple search of the files in the filesystem for any string matches to "dist/java-repository" would suffice. I checked it out and searched it on my laptop just now using Eclipse, there are no references in any of the files in any of geronimo. I been doing greps on cvs.apache.org and find the following cases as examples of references jakarta-turbine-2/project.properties,v:maven.repo.remote=http://www.ibiblio.org/maven/,http://www.apache.org/dist/java-repository/ This is a bad usage of java-repository, but not a show stopper, It will go to ibilio first then the ASF Repository, things should work fine as long as ibibilio's up. We should make sure there are no cases of the following for example: jakarta-turbine-2/project.properties,v:maven.repo.remote=http://www.apache.org/dist/java-repository/ Thats bad, theres really no reason this should be done at this point and time. Heres a really good example of a reference we promote usage of: jakarta-commons/chain/project.properties,v:maven.repo.central.directory=/www/www.apache.org/dist/java-repository maven.repo.central.directory is property which states "where" to publish to, not where to download from. It might be good to go through the Apache projects that are in java-repository and see if any are packaged to download dependencies from there, I doubt it that here would be any occurances however. As a majority of the files were originally published to/against ibiblio in the first place. Unfortunately, not all projects that use the apache repos are not hosted at apache. We've never publicized the the java-repository to be used externally. It 's content was originally migrated from Apache by Jason Van Zyl and Myself with the goal of getting publishing of Apache content back to Apache (with Ibibilio as a mirror). Anyone using it would be using it at their own risk. I believe that any consultation with the ASF Repository project here at Apache would have made this issue clear to them. Further, my point is that currently there shouldn't be projects dependent on java-repository as its location for resolving jar dependencies, there is an entire discussion on the [EMAIL PROTECTED] list concerning this subject, the repository used is a configuration of the client right now (in Maven) not a characteristic of the project.xml of a maven project. You can suggest a repository to use in properties handed to Maven, but the default is always www.ibiblio.org/maven. Our initial creation of java-repository was never not to have a "Standalone" Maven repository at Apache, but to provide a simple means for Apache developers to publish release jars and distributions to Ibibilio until such a time when distributed repositories become a greater practice. Its a real bad idea to force build dependencies against it at this time until we get it stabilized. Its entire contents are present at Ibiblio, there currently is no reason a project should be using it directly vs ibibilio. On top of this I recommend that Apache avoid maintaining interim builds in such a way that external projects can build dependencies against them. External projects should only be using fully 'sanctioned' releases published in dist for this purpose. Until tooling is completed in Ant and Maven to support distributing download requests from the ASF Repository across the mirrors we really cannot recommend that external projects build dependencies against the contents of the ASF Repository directly, please use
Re: Removing interim dated builds from /www/www.apache.org/dist/java-repository
On Jan 5, 2005, at 11:40 AM, Mark R. Diggory wrote: Dain Sundstrom wrote: Yes, I'm not sure that we can refer to it accepably as "a policy", as much as a "best practice". Well, if we take the latest release of Commons Math for instance. From both a Maven and Ant standpoint the distributions dependecies are resolved to/against ibibilio: http://www.ibiblio.org/maven/commons-logging/jars/commons- logging-1.0.3.jar"> http://www.ibiblio.org/maven/commons-discovery/jars/commons- discovery-0.2.jar"> I would hope so. These projects have a trivial or non existent set of dependencies. Take a look at a big project using maven and you will see that they use lots of repos, because ibiblio tends not to have everything necessary or was down when someone was working. In Maven, the repository used to get downloads from is not a project attiribute, it is a user decision/configuaration of the client. So really, a project that ends up with any dependency downloads directly from dist/java-repository is not the best solution, ibiblio is the more persistent and canonical point to be downloading from at this point. Take a look at geronimo and you will see that it is impractical for a user to hunt down dependencies. The code from a source distribution should just build, which means that all the dependencies *must* always be available. It might be good to go through the Apache projects that are in java-repository and see if any are packaged to download dependencies from there, I doubt it that here would be any occurances however. As a majority of the files were originally published to/against ibiblio in the first place. Unfortunately, not all projects that use the apache repos are not hosted at apache. -dain - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Removing interim dated builds from /www/www.apache.org/dist/java-repository
Dain Sundstrom wrote: On Jan 5, 2005, at 10:26 AM, Mark R. Diggory wrote: Thanks for your response Dain, Dain Sundstrom wrote: On Jan 4, 2005, at 8:34 PM, Mark R. Diggory wrote: Please excuse the cross post. I'm planning to run some commands on the java-repository to remove interim builds and SNAPSHOTS. Specifically, I'll be running: If you remove SNAPSHOTS, it not only can it break head branches of projects, Can you explain this further, I'm not sure what your suggesting here. Are you referring to content outside of /www/www.apache.org/dist/java-repository? Its important to point out that I'm only suggesting modification of content in the repository, not modification of any other content inside dist/... it can break milestone "source" releases that people are still using. -dain Well, the http://www.ibiblio.org/maven repository will still maintain all the SNAPSHOTS/interim builds I am planning on removing. That rsync does not delete files. So users of Maven working solely with the www.ibiblio.org/maven repository will not experience any of these changes. If it is really the case that Maven users are going to dist/java-repository instead www.ibiblio.org/maven we should alert them, dist/java-repository is in practice just for publishing Apache jars to ibiblio (though we have visions of its usage for other goals in the near future). For Apache developers using Maven, what I recommend is to continue using www.ibiblio.org to resolve dependencies for milestone and release builds. Then, to actually do any interim, snapshot or dated builds into http://cvs.apache.org/repository which is not mirrored. I can say that I was not aware of the policy, so I would assume other are not. If there are any projects that were using that location as a maven repo, their builds will break. Also if someone shipped a source distribution that needs one of these files, it will break, which means there would be no way to reproduce a historical build from a source distribution. -dain Yes, I'm not sure that we can refer to it accepably as "a policy", as much as a "best practice". Well, if we take the latest release of Commons Math for instance. From both a Maven and Ant standpoint the distributions dependecies are resolved to/against ibibilio: http://www.ibiblio.org/maven/commons-logging/jars/commons-logging-1.0.3.jar";> http://www.ibiblio.org/maven/commons-discovery/jars/commons-discovery-0.2.jar";> In Maven, the repository used to get downloads from is not a project attiribute, it is a user decision/configuaration of the client. So really, a project that ends up with any dependency downloads directly from dist/java-repository is not the best solution, ibiblio is the more persistent and canonical point to be downloading from at this point. It might be good to go through the Apache projects that are in java-repository and see if any are packaged to download dependencies from there, I doubt it that here would be any occurances however. As a majority of the files were originally published to/against ibiblio in the first place. cheers, MArk - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Removing interim dated builds from /www/www.apache.org/dist/java-repository
On Jan 5, 2005, at 10:26 AM, Mark R. Diggory wrote: Thanks for your response Dain, Dain Sundstrom wrote: On Jan 4, 2005, at 8:34 PM, Mark R. Diggory wrote: Please excuse the cross post. I'm planning to run some commands on the java-repository to remove interim builds and SNAPSHOTS. Specifically, I'll be running: If you remove SNAPSHOTS, it not only can it break head branches of projects, Can you explain this further, I'm not sure what your suggesting here. Are you referring to content outside of /www/www.apache.org/dist/java-repository? Its important to point out that I'm only suggesting modification of content in the repository, not modification of any other content inside dist/... it can break milestone "source" releases that people are still using. -dain Well, the http://www.ibiblio.org/maven repository will still maintain all the SNAPSHOTS/interim builds I am planning on removing. That rsync does not delete files. So users of Maven working solely with the www.ibiblio.org/maven repository will not experience any of these changes. If it is really the case that Maven users are going to dist/java-repository instead www.ibiblio.org/maven we should alert them, dist/java-repository is in practice just for publishing Apache jars to ibiblio (though we have visions of its usage for other goals in the near future). For Apache developers using Maven, what I recommend is to continue using www.ibiblio.org to resolve dependencies for milestone and release builds. Then, to actually do any interim, snapshot or dated builds into http://cvs.apache.org/repository which is not mirrored. I can say that I was not aware of the policy, so I would assume other are not. If there are any projects that were using that location as a maven repo, their builds will break. Also if someone shipped a source distribution that needs one of these files, it will break, which means there would be no way to reproduce a historical build from a source distribution. -dain - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Removing interim dated builds from /www/www.apache.org/dist/java-repository
Thanks for your response Dain, Dain Sundstrom wrote: On Jan 4, 2005, at 8:34 PM, Mark R. Diggory wrote: Please excuse the cross post. I'm planning to run some commands on the java-repository to remove interim builds and SNAPSHOTS. Specifically, I'll be running: If you remove SNAPSHOTS, it not only can it break head branches of projects, Can you explain this further, I'm not sure what your suggesting here. Are you referring to content outside of /www/www.apache.org/dist/java-repository? Its important to point out that I'm only suggesting modification of content in the repository, not modification of any other content inside dist/... it can break milestone "source" releases that people are still using. -dain Well, the http://www.ibiblio.org/maven repository will still maintain all the SNAPSHOTS/interim builds I am planning on removing. That rsync does not delete files. So users of Maven working solely with the www.ibiblio.org/maven repository will not experience any of these changes. If it is really the case that Maven users are going to dist/java-repository instead www.ibiblio.org/maven we should alert them, dist/java-repository is in practice just for publishing Apache jars to ibiblio (though we have visions of its usage for other goals in the near future). For Apache developers using Maven, what I recommend is to continue using www.ibiblio.org to resolve dependencies for milestone and release builds. Then, to actually do any interim, snapshot or dated builds into http://cvs.apache.org/repository which is not mirrored. My goal in cleaning the dist/java-repository directory is to eliminate the historical publishing of interim/dated builds from the java-repository so that we can begin to enforce the policy that these files should not be there via tool like Henk Pennings cron survey scripts: http://www.apache.org/~henkp/ -Mark -- Mark Diggory Software Developer Harvard MIT Data Center http://www.hmdc.harvard.edu - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Removing interim dated builds from /www/www.apache.org/dist/java-repository
On Jan 4, 2005, at 8:34 PM, Mark R. Diggory wrote: Please excuse the cross post. I'm planning to run some commands on the java-repository to remove interim builds and SNAPSHOTS. Specifically, I'll be running: If you remove SNAPSHOTS, it not only can it break head branches of projects, it can break milestone "source" releases that people are still using. -dain - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Removing interim dated builds from /www/www.apache.org/dist/java-repository
Please excuse the cross post. I'm planning to run some commands on the
java-repository to remove interim builds and SNAPSHOTS. Specifically,
I'll be running:
#!/bin/sh
> LOCATION=/www/www.apache.org/dist/java-repository
>
find ${LOCATION} -name '*200[0-4]*' | while read j;
do
rm -f $j;
done
find ${LOCATION} -name '*SNAPSHOT*' | while read j;
do
rm -f $j;
done
find ${LOCATION} -name '*snapshot-version*' | while read j;
do
rm -f $j;
done
I want to announce this so that folks have an opportunity to object or
make other recommendations.
For anyone wondering, all these files are currently available in
/www/archive.apache.org/dist/java-repository as well. So if anything
needs recovery it can be recovered from there.
-Mark
--
Mark Diggory
Open Source Software Developer
Apache Jakarta Project
http://jakarta.apache.org
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
