Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Tuesday 18 May 2010 12:59:28 William Kenworthy wrote: > On Tue, 2010-05-18 at 11:30 +0100, Peter Humphrey wrote: > > My setup does exactly the same, since squid is running on the same > > box. > > How have you configured it? - I wouldn't have though squid suitable > considering its designed for a different purpose and so regularly > expires items in its cache (i.e., they will be available for a > limited time before being cleaned.) If you extend max_age, then it > becomes unsuitable as a regular web proxy/cache unless you are > running multiple instances. There are posts saying that squid > doesnt work well with portage but other than a high miss rate > (possibly because the files expired?), no details are given. In view of what you say, maybe I ought to look into http-replicator. I have noticed some quite large files being fetched when I thought they ought already to be in squid's cache. -- Rgds Peter.
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Tue, 2010-05-18 at 11:30 +0100, Peter Humphrey wrote: > On Tuesday 18 May 2010 11:19:06 William Kenworthy wrote: > > > The advantage of http-replicator is that it is a caching proxy - if > > it isnt in the cache, it downloads it and then serves it out to one > > or more clients - rsync/FTP/wget/... can just share whats already > > there, not go get the file in the first place. > > My setup does exactly the same, since squid is running on the same box. > How have you configured it? - I wouldn't have though squid suitable considering its designed for a different purpose and so regularly expires items in its cache (i.e., they will be available for a limited time before being cleaned.) If you extend max_age, then it becomes unsuitable as a regular web proxy/cache unless you are running multiple instances. There are posts saying that squid doesnt work well with portage but other than a high miss rate (possibly because the files expired?), no details are given. Squid also seems to store its files named something like /var/cache/squid/00/00/00B9 so its hard to get at them directly without having squid to serve them up while the http-replicator cache is just the raw files - same as "distfiles" in fact. see http://forums.gentoo.org/viewtopic.php?p=1138287#1138287 for details on http-replicator. BillK -- William Kenworthy Home in Perth!
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Tuesday 18 May 2010 11:19:06 William Kenworthy wrote: > The advantage of http-replicator is that it is a caching proxy - if > it isnt in the cache, it downloads it and then serves it out to one > or more clients - rsync/FTP/wget/... can just share whats already > there, not go get the file in the first place. My setup does exactly the same, since squid is running on the same box. -- Rgds Peter.
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Tue, 18 May 2010 18:19:06 +0800, William Kenworthy wrote: > The advantage of http-replicator is that it is a caching proxy - if it > isnt in the cache, it downloads it and then serves it out to one or more > clients - rsync/FTP/wget/... can just share whats already there, not go > get the file in the first place. What happens if the proxy is not available, such as when a laptop is away from home? With NFS, the DISTDIR share simply isn't mounted and files are downloaded to the local directory, there's no configuration switch needed when away from home. Caching the files locally does have its advantages, but for me the only computer that would benefit from it, my netbook, is the one with the least storage space to spare. -- Neil Bothwick I'm not opinionated, I'm just always right! signature.asc Description: PGP signature
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Tue, 2010-05-18 at 10:12 +0100, Peter Humphrey wrote: > On Tuesday 18 May 2010 04:13:07 Bill Kenworthy wrote: > > > As an alternative check out http-replicator - yes the clients do > > download to a local directory but that can be cleaned afterwards. It > > also allows download locally when you know you are taking the machine > > (laptop?) elsewhere. > > Yet another approach is to have an rsync server on your LAN. In my case > it's the local server box (print, squid, mail etc). It's simple to set > up and to use, and all the boxes on the LAN can operate in their own > devious ways. > The advantage of http-replicator is that it is a caching proxy - if it isnt in the cache, it downloads it and then serves it out to one or more clients - rsync/FTP/wget/... can just share whats already there, not go get the file in the first place. BillK -- William Kenworthy Home in Perth!
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Tuesday 18 May 2010 04:13:07 Bill Kenworthy wrote: > As an alternative check out http-replicator - yes the clients do > download to a local directory but that can be cleaned afterwards. It > also allows download locally when you know you are taking the machine > (laptop?) elsewhere. Yet another approach is to have an rsync server on your LAN. In my case it's the local server box (print, squid, mail etc). It's simple to set up and to use, and all the boxes on the LAN can operate in their own devious ways. -- Rgds Peter.
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Mon, 2010-05-17 at 21:53 +0100, Neil Bothwick wrote: > On Mon, 17 May 2010 19:33:18 +0100, David W Noon wrote: > > > >I share my distfiles but I don't use FTP as that means storing copies > > >of the same file on each computer. Instead, I use NFS. /mnt/portage is > > >shared across all machines on the network and DISTDIR is set > > >to /mnt/portage/distfiles in each make.conf. > > > > I used to do that, but it meant my NFS server had to be running to > > perform any software maintenance on any box, so it became a single point > > of failure. The FTP approach allows each box to be self-reliant. > > Fair comment. I have DISTDIR on my mail server, so if that goes down, > I've more to worry about that a few tarballs. Even if it is inaccessible, > the other computers would simply download the files to the local > directory. > > > >Sharing /mnt/portage like this means I can also share my overlay across > > >the network at /mnt/portage/local. > > > > My boxes have different stuff in their overlays, and one uses no > > overlay packages at all. Sharing overlays doesn't make much sense for > > my set-up. > > It makes sense for me because everything is in one place, making > maintenance and backups simpler. Even if a package is only used on one > computer, for now, a central location still makes sense. > As an alternative check out http-replicator - yes the clients do download to a local directory but that can be cleaned afterwards. It also allows download locally when you know you are taking the machine (laptop?) elsewhere. An advantage over NFS is it seems to handle parallel downloads of the same file so you can transparently build all machines in parallel without the downloads stepping on each other over a common NFS mount. I also use a tmfs store for distfiles on one machine with plenty of ram so thats a self-cleaning (on reboot :) alternative. I have used NFS as well and its ok for data stores http-replicator is much better. Beware - NFS can be slow and flakey if used for building over (/var/tmp/portage). The great thing about gentoo's build system is its so flexible! BillK
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Mon, 17 May 2010 19:33:18 +0100, David W Noon wrote: > >I share my distfiles but I don't use FTP as that means storing copies > >of the same file on each computer. Instead, I use NFS. /mnt/portage is > >shared across all machines on the network and DISTDIR is set > >to /mnt/portage/distfiles in each make.conf. > > I used to do that, but it meant my NFS server had to be running to > perform any software maintenance on any box, so it became a single point > of failure. The FTP approach allows each box to be self-reliant. Fair comment. I have DISTDIR on my mail server, so if that goes down, I've more to worry about that a few tarballs. Even if it is inaccessible, the other computers would simply download the files to the local directory. > >Sharing /mnt/portage like this means I can also share my overlay across > >the network at /mnt/portage/local. > > My boxes have different stuff in their overlays, and one uses no > overlay packages at all. Sharing overlays doesn't make much sense for > my set-up. It makes sense for me because everything is in one place, making maintenance and backups simpler. Even if a package is only used on one computer, for now, a central location still makes sense. -- Neil Bothwick The computer revolution is over. The computers won. signature.asc Description: PGP signature
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Mon, 17 May 2010 13:50:02 +0200, Neil Bothwick wrote about Re: [gentoo-user] [SOLVED] identical drives, different free space!: >On Mon, 17 May 2010 12:31:17 +0100, David W Noon wrote: > >> >> Well, it turns out I have the distfiles mounted with --bind to my >> >> ftp/pub directory. And looking in the rsync man page: >> > >> >Why not set $DISTDIR to the true location of distfiles instead of >> >using bind mounts? >> >> Because binding the directory to /home/ftp/pub makes the distfiles >> available to the rest of one's network via anonymous ftp. I do the >> same thing here, without the "pub" subdirectory, and >> exclude /home/ftp/ from my backups. > >So the distfiles are actually in /usr/portage/distfiles? Correct. >I share my distfiles but I don't use FTP as that means storing copies >of the same file on each computer. Instead, I use NFS. /mnt/portage is >shared across all machines on the network and DISTDIR is set >to /mnt/portage/distfiles in each make.conf. I used to do that, but it meant my NFS server had to be running to perform any software maintenance on any box, so it became a single point of failure. The FTP approach allows each box to be self-reliant. >Sharing /mnt/portage like this means I can also share my overlay across >the network at /mnt/portage/local. My boxes have different stuff in their overlays, and one uses no overlay packages at all. Sharing overlays doesn't make much sense for my set-up. -- Regards, Dave [RLU #314465] == dwn...@ntlworld.com (David W Noon) == signature.asc Description: PGP signature
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Mon, 17 May 2010 21:50:28 +0930, Iain Buchanan wrote: > > I share my distfiles but I don't use FTP as that means storing copies > > of the same file on each computer. Instead, I use NFS. /mnt/portage is > > shared across all machines on the network and DISTDIR is set > > to /mnt/portage/distfiles in each make.conf. > Until I pick up my laptop and drive to work, where network speeds to my > server drop from 100Mbit to 50kbit and I need that local copy! I tend not to run emerges when away from home, although the lack of a local copy does prove awkward after a kernel upgrade that requires a rebuild of the wireless drivers. Not a situation I have to deal with any more, thankfully. > Which is why I'm glad there are multiple ways to do it :) Indeed :) -- Neil Bothwick A chicken is an egg's way of producing more eggs. signature.asc Description: PGP signature
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Mon, 2010-05-17 at 12:39 +0100, Neil Bothwick wrote: > On Mon, 17 May 2010 12:31:17 +0100, David W Noon wrote: ... > So the distfiles are actually in /usr/portage/distfiles? for me yes, it looks the same for David. > I share my distfiles but I don't use FTP as that means storing copies of > the same file on each computer. Instead, I use NFS. /mnt/portage is > shared across all machines on the network and DISTDIR is set > to /mnt/portage/distfiles in each make.conf. > > Sharing /mnt/portage like this means I can also share my overlay across > the network at /mnt/portage/local. Until I pick up my laptop and drive to work, where network speeds to my server drop from 100Mbit to 50kbit and I need that local copy! Which is why I'm glad there are multiple ways to do it :) -- Iain Buchanan Old robot: I choose to believe what I was programmed to believe.
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Mon, 2010-05-17 at 09:07 +0100, Neil Bothwick wrote: > On Mon, 17 May 2010 11:21:50 +0930, Iain Buchanan wrote: > > > Well, it turns out I have the distfiles mounted with --bind to my > > ftp/pub directory. And looking in the rsync man page: > > Why not set $DISTDIR to the true location of distfiles instead of using > bind mounts? because /usr/portage/distfiles IS the real location, and /home/ftp/pub/gentoo/distfiles is the ftp shared location. vsftpd doesn't handle symlinks, so I have to bind it. Now that you mention it though, I could move it for real into /home/ftp/pub/gentoo/distfiles and change DISTDIR... hm. -- Iain Buchanan Real programmers don't bring brown-bag lunches. If the vending machine doesn't sell it, they don't eat it. Vending machines don't sell quiche.
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Mon, 17 May 2010 12:31:17 +0100, David W Noon wrote: > >> Well, it turns out I have the distfiles mounted with --bind to my > >> ftp/pub directory. And looking in the rsync man page: > > > >Why not set $DISTDIR to the true location of distfiles instead of using > >bind mounts? > > Because binding the directory to /home/ftp/pub makes the distfiles > available to the rest of one's network via anonymous ftp. I do the > same thing here, without the "pub" subdirectory, and exclude /home/ftp/ > from my backups. So the distfiles are actually in /usr/portage/distfiles? I share my distfiles but I don't use FTP as that means storing copies of the same file on each computer. Instead, I use NFS. /mnt/portage is shared across all machines on the network and DISTDIR is set to /mnt/portage/distfiles in each make.conf. Sharing /mnt/portage like this means I can also share my overlay across the network at /mnt/portage/local. -- Neil Bothwick Top Oxymorons Number 18: Taped live signature.asc Description: PGP signature
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Mon, 17 May 2010 10:10:02 +0200, Neil Bothwick wrote about Re: [gentoo-user] [SOLVED] identical drives, different free space!: >On Mon, 17 May 2010 11:21:50 +0930, Iain Buchanan wrote: > >> Well, it turns out I have the distfiles mounted with --bind to my >> ftp/pub directory. And looking in the rsync man page: > >Why not set $DISTDIR to the true location of distfiles instead of using >bind mounts? Because binding the directory to /home/ftp/pub makes the distfiles available to the rest of one's network via anonymous ftp. I do the same thing here, without the "pub" subdirectory, and exclude /home/ftp/ from my backups. -- Regards, Dave [RLU #314465] == dwn...@ntlworld.com (David W Noon) == signature.asc Description: PGP signature
Re: [gentoo-user] [SOLVED] identical drives, different free space!
On Mon, 17 May 2010 11:21:50 +0930, Iain Buchanan wrote: > Well, it turns out I have the distfiles mounted with --bind to my > ftp/pub directory. And looking in the rsync man page: Why not set $DISTDIR to the true location of distfiles instead of using bind mounts? -- Neil Bothwick Tribble math: * + * = *** signature.asc Description: PGP signature
Re: [gentoo-user] [SOLVED] identical drives, different free space!
So after I excluded distfiles from my rsync, I found that the two partitions had roughly the same free space... strange! How could excluding around 6G of distfiles make two copies of the same thing the same size? Well, it turns out I have the distfiles mounted with --bind to my ftp/pub directory. And looking in the rsync man page: -x, --one-file-system ... Also keep in mind that rsync treats a "bind" mount to the same device as being on the same filesystem. So my distfiles were being copied in /usr/portage as well as /home/ftp/pub! Unfortunately the only way to get around it seems to be another --exclude directive. At least I understand what's going on now :) thanks for all the suggestions, -- Iain Buchanan Mr. Cole's Axiom: The sum of the intelligence on the planet is a constant; the population is growing.
