On Wed, Aug 12, 2009 at 10:51 PM, Robin H. Johnson <[email protected]>wrote:
> On Wed, Aug 12, 2009 at 05:17:55PM +0200, Francesco R wrote: > > Proposal, create snapshots of portage as squashfs iso, to be used in > > place of /usr/portage directory. > To all of these suggestions, I'd like to point out that if you're > willing to pay the same cost in administration (maintaining a separate > filesystem for /usr/portage), then you can have EVERYTHING in the > advantages list, and none of the things in the disadvantages list by > simply using a small reiserfs space for /usr/portage, with tail-packing > enabled. well squashfs has a pair of advantages over reiserfs, duplicate file detection, compression and a bright future. > > > For the rsync.g.o main rotation servers, we actually do that, just > RAM-backed to serve files as fast as possible without hitting disk. > only possibility to cope with the load they have I think > > When you removed bandwidth limitations and disk limitations on the > client side, I believe the record time for a emerge --sync that was 24 > hours out of date was somewhere around 23 seconds. 23 seconds are ... a lot without bandwidth and disk limitation, disk time for 50 MB is 1 sec (or even much less), and it transfer the whole iso in that time > > If you really wanted to get the rsync transfer size down, see what you > can do about the 'file list size' section, which is eating up a lot of > the download gains with the classical rsync:// sync. > IMHKnowledge the only way to do this is to have one index file (or files) the file should contain a triple ctime, status and file name (ordered by ctime possibly descending), and provide a cheap way to retrieve the list of files changed in a certain amount of time, status would be needed mainly for deleted files, but it can be modified or added too. Portage already has timestamp in /metadata so that skew of the client clock are not a problem, skew on the server would be. As a side advantage this could be served by an http server, still having rsync as an option. Currently rsync already use the option --whole-file and does only time/size check, if those differ, it downloat the full (little) file. Right ? This would be interesting too, but what happen if the timestamp in the client is too old or absent? fallback to rsync? How much time or how much size would the index file be allowed to grow? P.S. as a http client curl would be more useful than wget because it permit to download more file in one session
