Hi Peter,

Am 05.08.2012 um 15:42 schrieb peter green <plugw...@p10link.net>:
> I am setting up mirrorbrain to manage mirrors for the apt repository for 
> raspbian (a rebuild of debian for armv6 hardfloat).
> In my setup the server running mirrorbrain is also the public master server 
> (it in-turn is updated from the private master server but that is not 
> relavent here). This means that any changes will (or at least should) happen 
> on the master server first.
> Package updates come in the form of additions and removals of files which 
> mirrorbrain seems to have no problem with. However the package lists and 
> related files do get modified in-place.
> What I would like to happen is when a file is changed locally I would like 
> mirrorbrain to stop redirecting it until the mirrors have been rescanned and 
> confirmed as having the new version of the file. In this way I can ensure 
> that users always get the latest version of the package lists regardless of 
> whether the mirrors are up to date.
> Currently mirrorbrain seems to have some knowlege of file dates (it shows the 
> updated date on the mirror list page for the file) but it seems to carry on 
> redirecting clients regardless of the changed file date of the local copy.
> Currently my workarround is to exclude the dists dir from mirrorbrain, 
> however I'd rather not do this if I can help it.

Looking around in the tree, I see several files named "Packages" or "Release", 
thus with no versioning indication in their name, and on the other hand, 
package files with version info (including build or release numbering). The 
latter files are easy to handle for MirrorBrain, because a file is assumed to 
never change - if a package is rebuilt, the release counter is incremented. The 
"version-less" files however are more difficult to handle. MirrorBrain doesn't 
store modification times or file size for what it finds on mirrors. (It would 
have been possible to implement it that way, but it was decided against for 
performance reasons [which might not affect some users in fact...].) 

There are some ways to deal with it: 

If the files are small, simply don't redirect for them; just deliver them. You 
can use the MirrorBrainExcludeFileMask directive with a regexp, e.g. 
"\.(xml|asc)". This can save the client an additional roundtrip to the mirror. 
And for certain files, like crypto-hashes and signatures, it makes sense to 
deliver them directly as well, for security reasons.

For larger files, a workaround is required, so the load can be offloaded to 
mirrors by redirection. Say you have a file named "Packages" which is updated 
frequently. You could rename the file to make its name unique. It would be 
named e.g. "Packages-${sha1sum}", or "Packages-${timestamp}". A symlink named 
"Packages" would point to the current file. Whenever the file is updated, you 
update the symlink. This works because MirrorBrain resolves links before 
considering redirection. Therefore, MirrorBrain would send the client only to 
mirrors that have the current file. It wouldn't matter anymore if there are 
files names "Packages" on the mirrors; only the current uniquely named file 
would be downloaded. 

http://mirrors.xbmc.org/nightlies/osx/ppc/ is an example of this technique.

When mirrors sync and get the new uniquely named files, and you scan them, 
redirection to them will commence. To scan subdirectories, the "mb scan -d 
<dir>" command can be very useful (saving a full scan).

If it is important to spread files quickly, to save load, mirrors either need 
to sync very frequently (and need to be scanned frequently), or the content 
need to be pushed (which gives better control, and one knows exactly when to 
scan the mirrors). This little script
is used by some people to implement content pushing to mirrors in parallel. It 
requires write access on mirrors. Otherwise, triggered pull syncing is another 
possible way to implement this (as Debian does). 

Does this help?


mirrorbrain mailing list
Archive: http://mirrorbrain.org/archive/mirrorbrain/

Note: To remove yourself from this mailing list, send a mail with the content
to the address mirrorbrain-requ...@mirrorbrain.org

Reply via email to