Re: closer (was Re: TODOs)
Adam R. B. Jack wrote: "Mark R. Diggory" <[EMAIL PROTECTED]> wrote: ASF Repository: About this time what I'm maintaining is the following two repository directories: Thanks for this write-up. I haven't yet explored the idea of getting a "repository.apache.org" virtual host going. Up to this point I wanted to see the reuse of the existing mirroring structure to get these artifacts out to multiple hosts. I know that Henri and others have been working on some new download pages and scripts for redirecting to mirrors for downloads. I'd be really interested in finding out how we can combine this sort of Metadata with a download client to get clients downloading "load balanced" across the mirrors. I suspect this would be a server side redirect mechanism of some sort. In Depot Update we've tinkered with picking one based off this URL's contents. http://www.apache.org/dyn/closer.cgi/java-repository/ Are you talking about closer.cgi, or a newer script? regards Adam Hmm, I was reviewing the emails and most of it was about building proper links to closer.cgi from the various download pages in Jakarta Commons. Not neccessarily improving upon closer.cgi. http://www.mail-archive.com/[email protected]/msg43827.html I think what really needs to happen is that theres a "machine" readable or parametized version of this script which returns either a redirect or a machine parsable list of locations (for instance in XML/RDF) instead of the human readable html page. The current script doesn't offer this, but a "template" could written for it to meet this need. At first I thought it would be fine to just have a serverside script that returns a redirect. But after some thought I realized the client in this situation would probibly just be a simple Http client or ftp client and if the connection failed, there would be a need for some sort of mechanism for recovery and retry at a new location, in which case getting and parsing a list would give the client multiple locations to choose from. I think this could really be accomplished by extending the script as such that it could detect user-agent or other request header/parameter and return the appropriate formatted content instead. Then the client would parse the content and iterate over the list until it retrieved a successfull download. -Mark begin:vcard fn:Mark Diggory n:Diggory;Mark org:Harvard University;Harvard MIT Data Center adr:Harvard University;;G-6 Littauer Center (North Yard);Cambridge;Ma;02138-2901;United States email;internet:[EMAIL PROTECTED] title:Software Engineer tel;work:617 496 7246 tel;fax:617 495 0438 tel;home:617 718 2033 tel;cell:617 285 4106 url:http://www.hmdc.harvard.edu version:2.1 end:vcard
closer (was Re: TODOs)
"Mark R. Diggory" <[EMAIL PROTECTED]> wrote: > ASF Repository: > > About this time what I'm maintaining is the following two repository > directories: Thanks for this write-up. > I haven't yet explored the idea of getting a "repository.apache.org" > virtual host going. Up to this point I wanted to see the reuse of the > existing mirroring structure to get these artifacts out to multiple > hosts. I know that Henri and others have been working on some new > download pages and scripts for redirecting to mirrors for downloads. I'd > be really interested in finding out how we can combine this sort of > Metadata with a download client to get clients downloading "load > balanced" across the mirrors. I suspect this would be a server side > redirect mechanism of some sort. In Depot Update we've tinkered with picking one based off this URL's contents. http://www.apache.org/dyn/closer.cgi/java-repository/ Are you talking about closer.cgi, or a newer script? regards Adam
Re: TODOs
Adam R. B. Jack wrote: Ok, time to re-gen a set of todos since we have a few folks who are able to scratch this itch right now. Here are some things that come to my mind. 1) ASF Repository Find out more about what has been implemented in the name of the ASF Repository, and report to this list. I know something called this exists somewhere (although I don't think it follows an ASF repo layout, but a Maven one) I'd like to poke it, prod it, test it. ASF Repository: About this time what I'm maintaining is the following two repository directories: Production Release Repository: This repository is located in dist and mirrored to all Apache Mirrors. This is a mission critical repository for Apache, we maintain all Apache artifacts being released for public consumption here and it is rsynced to the www.ibiblio.org/maven directory every 4 hours overwriting anything updated. This allows us to manage the Apache content in the Maven repository locally which is more acceptable because we have fully control over when it is released into dist at Apache. /www/www.apache.org/dist/java-repository http://www.apache.org/dist/java-repository Interm/Nightly/Beta Repository: This is a Maven repository outside the dist directory for artifacts we do not want to see distributed to mirrors or anywhere else for that matter. This is a repository for developers to test deployment into and to "release" developmental builds to the "Apache Developer Community". /www/cvs.apache.org/repository http://cvs.apache.org/repository I haven't yet explored the idea of getting a "repository.apache.org" virtual host going. Up to this point I wanted to see the reuse of the existing mirroring structure to get these artifacts out to multiple hosts. I know that Henri and others have been working on some new download pages and scripts for redirecting to mirrors for downloads. I'd be really interested in finding out how we can combine this sort of Metadata with a download client to get clients downloading "load balanced" across the mirrors. I suspect this would be a server side redirect mechanism of some sort. -Mark 2) Repository/Artifact (Avalon interface) This re-work was very close, needs to be completed (and the Ant Type needs to be re-tested over it). One for me, since I started it... 3) Downloading Create a separate DownloadManager that does downloads for us, and does things like check MD5s, and allows user-set listeners for doing things like '.' (dots) for downloaded blocks, or percentages in GUIs, etc. 4) API Keep refactoring/documenting to generate a simple API. A 'hands on' interface, for those wanting to manually work w/ Repository/Artifacts (Avaloners) and an Updater one (for querying repositories and downloading latest best fit). 5) Monitor the Gump errors and fix what needs fixing. regards, Adam -- Experience the Unwired Enterprise: http://www.sybase.com/unwiredenterprise Try Sybase: http://www.try.sybase.com begin:vcard fn:Mark Diggory n:Diggory;Mark org:Harvard University;Harvard MIT Data Center adr:Harvard University;;G-6 Littauer Center (North Yard);Cambridge;Ma;02138-2901;United States email;internet:[EMAIL PROTECTED] title:Software Engineer tel;work:617 496 7246 tel;fax:617 495 0438 tel;home:617 718 2033 tel;cell:617 285 4106 url:http://www.hmdc.harvard.edu version:2.1 end:vcard
TODOs
Ok, time to re-gen a set of todos since we have a few folks who are able to scratch this itch right now. Here are some things that come to my mind. 1) ASF Repository Find out more about what has been implemented in the name of the ASF Repository, and report to this list. I know something called this exists somewhere (although I don't think it follows an ASF repo layout, but a Maven one) I'd like to poke it, prod it, test it. 2) Repository/Artifact (Avalon interface) This re-work was very close, needs to be completed (and the Ant Type needs to be re-tested over it). One for me, since I started it... 3) Downloading Create a separate DownloadManager that does downloads for us, and does things like check MD5s, and allows user-set listeners for doing things like '.' (dots) for downloaded blocks, or percentages in GUIs, etc. 4) API Keep refactoring/documenting to generate a simple API. A 'hands on' interface, for those wanting to manually work w/ Repository/Artifacts (Avaloners) and an Updater one (for querying repositories and downloading latest best fit). 5) Monitor the Gump errors and fix what needs fixing. regards, Adam -- Experience the Unwired Enterprise: http://www.sybase.com/unwiredenterprise Try Sybase: http://www.try.sybase.com
