On Thu, Apr 17, 2008 at 07:15:03AM +1000, Russell Dickenson <[EMAIL PROTECTED]> wrote: > Recently the Frugalware mirror closest to me - ftp13 - became > unreliable. When pacman-g2 repeatedly failed to download packages > from this mirror it had me thinking about the problem.
side note: the mirror is up again and seem to be up to date as well. > Here I need to add a standard disclaimer. What I'm about to describe > may already have been thought of, discussed and dismissed as being a > "bad idea". Anyway, here are my thoughts for whatever they're worth. > I should point out that the Australian dollar is just a few cents less > in value than the USA dollar. :) your suggestions are always welcome ;) > Here's how I understand Frugalware package mirroring to work, and how > releases are affected - > > Around the time of a new Frugalware release, a new folder is created > with its name being the release's version number. Into that folder > (or perhaps a sub-folder) go all the packages for that release. These > initially go to the master server and are then mirrored by the various > Frugalware mirrors. Mirroring takes some time so there's a few days > of waiting between the release and the release's announcement. Once > the announcement is made, people start downloading and installing or > upgrading. to be correct we wait 24h since we require every mirror to sync at least daily. however, this was usually not required since all mirror was synced at least two times a day. the problem you mention is - i think - mainly about we're getting more and more mirrors, so it's more regular that one of them has problems. with the last release this was more problematic, actually our default mirror (ftp5) went down, so everyone got warnings before pacman tried to connect to the next server. > Some mirrors may still be out of sync with the master and > so pacman-g2 tries all the mirrors in its list for each package. > > Now for my idea (finally!) - why not have one folder named "packages" > (or something similar) which contains all packages for current and the > latest release? When the following release occurs, most of the > packages required will have been in current for some time and so > already be mirrored. At most there will be a relatively small number > of packages, mostly to do with release naming, configuration files > etc, still to be mirrored. If this method was used I believe it would > reduce the time it takes each mirror to be sync-ed for each new > release. the idea is good, but i think it's unrelated to the problem. even with that big amount of data we push out with a new release (-0.8 is currently 23GB), a mirror is in sync or is not in sync, but the "sync is in progress" state is minimal. so the problem is not about "hey, that mirror will be in sync only after 2 days as it's slow" but "hey, that mirror is down, or it's up but it's no longer updated at all". hardlinking the packages may make sense as rsync transfers hardlinks, but this is not used by all mirrors, for example if they use afs, afs does not support hardlinks (afs is a distributed filesystem frequently used by mirrors). also, if we want to use hardlinks, we have to check our scripts, for example the changelogs are generated like 'git log filename > Changelog' at the moment, using hardlinks would mean - in this case - that the stable and current tree would not have separate changelog files which is bad. (changing the script to 'rm -f Changelog; git log filename >Changelog' would help). what may be a good idea is to set up some monitoring for mirrors. i just went through the mirrors on frugalware.org/download and created a small stat: - unreachable: ftp6 - outdated: ftp9 - up to date: the rest (8 mirrors) i think that amount is not that bad :) though, it would be nice to have some automated test for this. i just found a package: http://people.cs.uu.nl/henkp/mirmon/ i'll play with it a bit :)
pgptxGrt5w10b.pgp
Description: PGP signature
_______________________________________________ Frugalware-devel mailing list [email protected] http://frugalware.org/mailman/listinfo/frugalware-devel
