Re: [gentoo-user] Best file system for portage tree?
On Mon, Mar 12, 2012 at 10:23:49AM +0100, Alex Schuster wrote: José Romildo Malaquias writes: On Sat, Mar 10, 2012 at 07:36:07PM +0100, YoYo Siska wrote: mke2fs -f -b1024 -i2048 /usr/img_portage The -f option from mke2fs is to specify a fragment size and expects an argument. Do you -F (which forces mke2fs to create a filesystem, even if the specified device is not a partittion on a block special device)? I'm pretty sure that's what he meant, without the -F you need to confirm that you really want to create the FS. I forgot to mention this when I also quoted this line in my reply. yes, that's exactly what I had in mind was writing that from memory and forgot the the case... thanks for the correction yoyo
Re: [gentoo-user] Best file system for portage tree?
On Sat, Mar 10, 2012 at 07:36:07PM +0100, YoYo Siska wrote: On Sat, Mar 10, 2012 at 03:35:05PM +, Neil Bothwick wrote: On Sat, 10 Mar 2012 14:30:15 +0100, Alex Schuster wrote: Any tips on this? Does it make sense to use a special file system just for the portage tree? What would be best? Would it help to re-create this file system from time to time in case it gets slower with every sync? I use an ext2 filesystem for portage, it's still the fastest out there. Journals are unnecessary because its such a small filesystem, and if it does get damaged I can just reformat and sync again. I use an ext2 partition in a 500MB file image on most of my computers. Its important to check the inode count on such small filesytem, as mke2fs' default inode ration for such size is 4096, which is too low for portage: dd bs=$((500*1024*1024)) count=1 if=/dev/zero of=/usr/img_portage mke2fs -f -b1024 -i2048 /usr/img_portage The -f option from mke2fs is to specify a fragment size and expects an argument. Do you -F (which forces mke2fs to create a filesystem, even if the specified device is not a partittion on a block special device)? fstab: /usr/img_portage/usr/portage/ ext2loop,noatime 0 0 (this is from desktop, on servers I usually only mount it manually when emerging) # df -h Filesystem Size Used Avail Use% Mounted on /dev/loop0 469M 306M 139M 69% /usr/portage # df -i FilesystemInodes IUsed IFree IUse% Mounted on /dev/loop0256032 152044103988 60% /usr/portage Romildo
Re: [gentoo-user] Best file system for portage tree?
José Romildo Malaquias writes: On Sat, Mar 10, 2012 at 07:36:07PM +0100, YoYo Siska wrote: mke2fs -f -b1024 -i2048 /usr/img_portage The -f option from mke2fs is to specify a fragment size and expects an argument. Do you -F (which forces mke2fs to create a filesystem, even if the specified device is not a partittion on a block special device)? I'm pretty sure that's what he meant, without the -F you need to confirm that you really want to create the FS. I forgot to mention this when I also quoted this line in my reply. Wonko
Re: [gentoo-user] Best file system for portage tree?
YoYo Siska writes: On Sat, Mar 10, 2012 at 03:35:05PM +, Neil Bothwick wrote: I use an ext2 filesystem for portage, it's still the fastest out there. Journals are unnecessary because its such a small filesystem, and if it does get damaged I can just reformat and sync again. Replaying a reiserfs journal in case of an unclean reboot also takes about the same time as an whole e2fsck, so I switched to ext2. There was no real need to make the switch, I just wanted to re-create this file system that has been synced very often now. I use an ext2 partition in a 500MB file image on most of my computers. I also did this in the past, on systems where I did not use LVM. Nowadays I prefer the latter. Its important to check the inode count on such small filesytem, as mke2fs' default inode ration for such size is 4096, which is too low for portage: Yes, happened to me more than once... mke2fs -f -b1024 -i2048 /usr/img_portage That's what I did. Well, without the container file. Thanks to all who replied! I learnt something, like so often when reading here. Wonko
[gentoo-user] Best file system for portage tree?
Hi there! Is there an advantage in putting the portage tree on an extra partition? Currently, I'm using reiserfs, because I read that it is efficient when using many small files. On the other hand I also heard that it tends to get slower with every emerge --sync. Space is no longer an argument in these days, at least for my desktop machine. But I would like to optimize for speed -- emerge -DputnVj @world takes quite a while to calculate, I assume this is because so many ebuild files have to be accessed. Any tips on this? Does it make sense to use a special file system just for the portage tree? What would be best? Would it help to re-create this file system from time to time in case it gets slower with every sync? Or wouldn't I notice a difference if I just used a big ext4 partition for all portage related stuff? Anyone using a compressed RAM file system for that? :) Wonko
Re: [gentoo-user] Best file system for portage tree?
Alex Schuster wrote: Hi there! Is there an advantage in putting the portage tree on an extra partition? Currently, I'm using reiserfs, because I read that it is efficient when using many small files. On the other hand I also heard that it tends to get slower with every emerge --sync. Space is no longer an argument in these days, at least for my desktop machine. But I would like to optimize for speed -- emerge -DputnVj @world takes quite a while to calculate, I assume this is because so many ebuild files have to be accessed. Any tips on this? Does it make sense to use a special file system just for the portage tree? What would be best? Would it help to re-create this file system from time to time in case it gets slower with every sync? Or wouldn't I notice a difference if I just used a big ext4 partition for all portage related stuff? Anyone using a compressed RAM file system for that? :) Wonko I have mine on its own partition. Faster, not sure but most likely. I use ext3 for mine. Since I am redoing my partitions, I'm looking forward to reading what others say. Dale :-) :-) -- I am only responsible for what I said ... Not for what you understood or how you interpreted my words! Miss the compile output? Hint: EMERGE_DEFAULT_OPTS=--quiet-build=n
Re: [gentoo-user] Best file system for portage tree?
Am 10.03.2012 14:30, schrieb Alex Schuster: Hi there! Is there an advantage in putting the portage tree on an extra partition? Yes. It allows you to use a smaller and more appropriate block size like 1k or 2k which decreases internal fragmentation. It also increases locality of data, meaning that you won't scatter your files all over your 2TB hard disk. Ext* and co. have mechanisms to prevent this but it still helps to enforce it. Currently, I'm using reiserfs, because I read that it is efficient when using many small files. On the other hand I also heard that it tends to get slower with every emerge --sync. Yes, that's a problem of every file system. Reiserfs (especially without notail) and btrfs are more prone to this due to their internal organization. Space is no longer an argument in these days, at least for my desktop machine. But I would like to optimize for speed -- emerge -DputnVj @world takes quite a while to calculate, I assume this is because so many ebuild files have to be accessed. Not just ebuilds. You also have to consider /var/cache/edb and /var/db/pkg. Be careful with the latter one. You don't want to loose its content. Any tips on this? Does it make sense to use a special file system just for the portage tree? What would be best? Would it help to re-create this file system from time to time in case it gets slower with every sync? Or wouldn't I notice a difference if I just used a big ext4 partition for all portage related stuff? Anyone using a compressed RAM file system for that? :) Wonko Recreating it certainly helps. I don't find it worth the effort. though. Regards, Florian Philipp signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] Best file system for portage tree?
On Mar 10, 2012 8:33 PM, Alex Schuster wo...@wonkology.org wrote: Hi there! Is there an advantage in putting the portage tree on an extra partition? Currently, I'm using reiserfs, because I read that it is efficient when using many small files. On the other hand I also heard that it tends to get slower with every emerge --sync. Space is no longer an argument in these days, at least for my desktop machine. But I would like to optimize for speed -- emerge -DputnVj @world takes quite a while to calculate, I assume this is because so many ebuild files have to be accessed. Any tips on this? Does it make sense to use a special file system just for the portage tree? What would be best? Would it help to re-create this file system from time to time in case it gets slower with every sync? Or wouldn't I notice a difference if I just used a big ext4 partition for all portage related stuff? Anyone using a compressed RAM file system for that? :) This had been my burning question when I was deploying the company's production server, and forced me to do some research: * reiserfs is amazingly fast for reads, but suffers on simultaneous writes * reiserfs does not have inode limits * reiserfs' notail affects performance greatly depending on the nature of the system: I/O-bound (use notail) or CPU-bound (don't use notail) * reiserfs, if mounted without notail, is very space-efficient So, I end up with the following mix: * ext2 for /boot * reiserfs for /usr/portage and /var/tmp (RAM is at premium; can't use tmpfs) * ext4 for everything else This cocktail has been serving me well. I don't need advanced filesystems like ZFS, XFS, or btrfs, because my servers are virtualized, and the advanced features (e.g., snapshot) is handled by the underlying hypervisor (XenServer) and SAN Storage (we use NetApp). Rgds,
Re: [gentoo-user] Best file system for portage tree?
On Mar 10, 2012 10:09 PM, Pandu Poluan pa...@poluan.info wrote: On Mar 10, 2012 8:33 PM, Alex Schuster wo...@wonkology.org wrote: Hi there! Is there an advantage in putting the portage tree on an extra partition? Currently, I'm using reiserfs, because I read that it is efficient when using many small files. On the other hand I also heard that it tends to get slower with every emerge --sync. Space is no longer an argument in these days, at least for my desktop machine. But I would like to optimize for speed -- emerge -DputnVj @world takes quite a while to calculate, I assume this is because so many ebuild files have to be accessed. Any tips on this? Does it make sense to use a special file system just for the portage tree? What would be best? Would it help to re-create this file system from time to time in case it gets slower with every sync? Or wouldn't I notice a difference if I just used a big ext4 partition for all portage related stuff? Anyone using a compressed RAM file system for that? :) This had been my burning question when I was deploying the company's production server, and forced me to do some research: * reiserfs is amazingly fast for reads, but suffers on simultaneous writes * reiserfs does not have inode limits * reiserfs' notail affects performance greatly depending on the nature of the system: I/O-bound (use notail) or CPU-bound (don't use notail) * reiserfs, if mounted without notail, is very space-efficient So, I end up with the following mix: * ext2 for /boot * reiserfs for /usr/portage and /var/tmp (RAM is at premium; can't use tmpfs) * ext4 for everything else This cocktail has been serving me well. I don't need advanced filesystems like ZFS, XFS, or btrfs, because my servers are virtualized, and the advanced features (e.g., snapshot) is handled by the underlying hypervisor (XenServer) and SAN Storage (we use NetApp). Rgds, Okay, I did a mixup: If the system is I/O-bound, *don't* use notail (saves on disk read/write). If the system is CPU-bound, *use* notail (saves on having to 'unpack' the tail from the metadata). In my situation, the bottleneck is the SAN Storage, so I don't use notail. Rgds,
Re: [gentoo-user] Best file system for portage tree?
On Sat, 10 Mar 2012 14:30:15 +0100, Alex Schuster wrote: Any tips on this? Does it make sense to use a special file system just for the portage tree? What would be best? Would it help to re-create this file system from time to time in case it gets slower with every sync? I use an ext2 filesystem for portage, it's still the fastest out there. Journals are unnecessary because its such a small filesystem, and if it does get damaged I can just reformat and sync again. -- Neil Bothwick Physics is like sex: sure, it may give some practical results, but that's not why we do it.Richard Feynman signature.asc Description: PGP signature
Re: [gentoo-user] Best file system for portage tree?
On Sat, 10 Mar 2012 22:09:26 +0700 Pandu Poluan pa...@poluan.info wrote: On Mar 10, 2012 8:33 PM, Alex Schuster wo...@wonkology.org wrote: Hi there! Is there an advantage in putting the portage tree on an extra partition? Currently, I'm using reiserfs, because I read that it is efficient when using many small files. On the other hand I also heard that it tends to get slower with every emerge --sync. Space is no longer an argument in these days, at least for my desktop machine. But I would like to optimize for speed -- emerge -DputnVj @world takes quite a while to calculate, I assume this is because so many ebuild files have to be accessed. Any tips on this? Does it make sense to use a special file system just for the portage tree? What would be best? Would it help to re-create this file system from time to time in case it gets slower with every sync? Or wouldn't I notice a difference if I just used a big ext4 partition for all portage related stuff? Anyone using a compressed RAM file system for that? :) This had been my burning question when I was deploying the company's production server, and forced me to do some research: * reiserfs is amazingly fast for reads, but suffers on simultaneous writes * reiserfs does not have inode limits * reiserfs' notail affects performance greatly depending on the nature of the system: I/O-bound (use notail) or CPU-bound (don't use notail) * reiserfs, if mounted without notail, is very space-efficient So, I end up with the following mix: * ext2 for /boot * reiserfs for /usr/portage and /var/tmp (RAM is at premium; can't use tmpfs) * ext4 for everything else This cocktail has been serving me well. I don't need advanced filesystems like ZFS, XFS, or btrfs, because my servers are virtualized, and the advanced features (e.g., snapshot) is handled by the underlying hypervisor (XenServer) and SAN Storage (we use NetApp). Rgds, That's very close to what I do (though not for the same extensively-researched reasons :). I added an extra bit of twiddling in make.conf: DISTDIR=/usr/local/distfiles # On /. PKGDIR=/usr/local/packages # On /. PORTDIR=/mnt/portage/gentoo # /mnt/portage is reiserfs and has /layman too This way the requirements for the portage partition grow much more gradually (changed that due to overflow once), and on the random chance that reiserfs gets corrupted, I don't lose all my fetch-restricted distfiles. - Bryan
Re: [gentoo-user] Best file system for portage tree?
On Sat, Mar 10, 2012 at 03:35:05PM +, Neil Bothwick wrote: On Sat, 10 Mar 2012 14:30:15 +0100, Alex Schuster wrote: Any tips on this? Does it make sense to use a special file system just for the portage tree? What would be best? Would it help to re-create this file system from time to time in case it gets slower with every sync? I use an ext2 filesystem for portage, it's still the fastest out there. Journals are unnecessary because its such a small filesystem, and if it does get damaged I can just reformat and sync again. I use an ext2 partition in a 500MB file image on most of my computers. Its important to check the inode count on such small filesytem, as mke2fs' default inode ration for such size is 4096, which is too low for portage: dd bs=$((500*1024*1024)) count=1 if=/dev/zero of=/usr/img_portage mke2fs -f -b1024 -i2048 /usr/img_portage fstab: /usr/img_portage/usr/portage/ ext2loop,noatime 0 0 (this is from desktop, on servers I usually only mount it manually when emerging) # df -h Filesystem Size Used Avail Use% Mounted on /dev/loop0 469M 306M 139M 69% /usr/portage # df -i FilesystemInodes IUsed IFree IUse% Mounted on /dev/loop0256032 152044103988 60% /usr/portage yoyo
Re: [gentoo-user] Best file system for portage tree?
On Sat, 10 Mar 2012 19:36:07 +0100, YoYo Siska wrote: I use an ext2 filesystem for portage, it's still the fastest out there. Journals are unnecessary because its such a small filesystem, and if it does get damaged I can just reformat and sync again. I use an ext2 partition in a 500MB file image on most of my computers. I used to do that but, after switching to LVM, it was simpler to use an LV. Its important to check the inode count on such small filesytem, as mke2fs' default inode ration for such size is 4096, which is too low for portage: dd bs=$((500*1024*1024)) count=1 if=/dev/zero of=/usr/img_portage mke2fs -f -b1024 -i2048 /usr/img_portage I use similar arguments for mke2fs. -- Neil Bothwick Why do they call it a TV set when you only get one? signature.asc Description: PGP signature
