On Mon, Nov 2, 2009 at 11:33 AM, Shawn Walker <[email protected]> wrote: > Mike Gerdts wrote: >> >> Andrzej Szeszo has posted a script that seems to address all of these >> issues. Is there a reason that this approach is bad? >> >> http://aszeszo.blogspot.com/2009/07/ips-repository-mirror-script_27.html >> >> It seems to me that the following (adapted from an example in --help) >> would put just a little more stress on the opensolaris.org/dev repo >> than "pkg install redistributable" from a fresh install. > > So, there are several things about this program that aren't quite right: > > * doesn't use pkg client api, so "latest" version determination isn't > reliable; also makes assumptions about package version which means this will > only work for /dev and /release, and may not at some future point > > * doesn't use a modern transport api, so won't download manifests or catalog > data using compressed transport (which means it will be slower than the > actual pkg(1) client) > > * is using /catalog/0/, which means it won't be able to handle certain > repository configurations (multi-publisher content) > > * is using old repository directory structure, which also just recently > changed (though this will still work with the new depot server) > > * will only scale very well to a single build; can't imagine using it to > maintain a full /dev mirror > > * assumes a certain structure for the repository and may break at some > future point > > You should also be aware that the depot software and client currently assume > that all files declared in a package manifest exist in the repository. > While it shouldn't cause any problems at the moment to only have SPARC or > i386 data in your repository, it is not officially supported. > > However, from a strictly "does it harm the package server" standpoint, not > really. Although again, the package servers are not currently provisioned > for mirroring using mechanisms other than rsync, and we continue to request > that you pull at 2AM your local time. > > Work continues to improve the performance of the existing mirroring options > and to provide additional ones.
Thanks for the feedback. At the current time I don't feel like I need a mirror - rather I need to not have to refetch everything every time that another box installs or I test a new install on the same box. What obstacles (other than implementation time/effort) would there be to using a caching reverse proxy? While this could be in the form of an off-the-shelf product like Apache's mod_proxy, I suspect that an extension to the existing package server would be beneficial for many of the reasons you state above. The way I would foresee this proxy package server working is as follows. - Various images would point to the proxy package server as the origin_uri - When the local proxy package server sees a request for a manifest or file it hasn't already cached, it downloads and caches it. - All communication between the proxy package server and the primary package server uses the pkg API - The proxy package server could be configured to prune cached items that are no in high demand - If someone was to warm the cache they can do so with pkg image-create + pkg install to scratch space. Not only would I see this as useful for caching the opensolaris.org repository, but it would seem to be a good mechanism for just-in-time replication of the relevant portions of repositories in my global enterprise. Is there a better tree to bark up? If I produced working code that did this, would there be interest in it? -- Mike Gerdts http://mgerdts.blogspot.com/ _______________________________________________ pkg-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
