[gentoo-user] Re: Fragmentation of my drives. Curious mostly
Nikos Chantziaras wrote: Jorge Peixoto de Morais Neto wrote: [...] what would be the best way to defrag it? By not defragging it. [...] I don't buy into that argument and never did. Every few months I copy the whole HD to another one and then back to counter fragmentation (ext3) and the system becomes noticeably faster after doing it (speed increase in emerge --sync for example.) Maybe it's not fragmentation but rather related files being more closely together after I do this. How exactly do you copy the files? [...] I simply boot from the Gentoo DVD and rsync to another ext3 partition, wipe the current filesystem and then rsync back. OK, I once again verified that fragmentation seems to be a big issue even on Linux. I just migrated to ext4, and in order to do that I had to rsync, format and rsync back. The result is similar to the last time I did this (over 8 months ago): emerge --sync takes 15 seconds (at least 3 minutes yesterday) update-eix takes 2 seconds (20 seconds yesterday) And I don't believe it's due to ext4. It's a nice speed-up from ext3, but not THAT nice.
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Nikos Chantziaras wrote: Nikos Chantziaras wrote: Jorge Peixoto de Morais Neto wrote: [...] what would be the best way to defrag it? By not defragging it. [...] I don't buy into that argument and never did. Every few months I copy the whole HD to another one and then back to counter fragmentation (ext3) and the system becomes noticeably faster after doing it (speed increase in emerge --sync for example.) Maybe it's not fragmentation but rather related files being more closely together after I do this. How exactly do you copy the files? [...] I simply boot from the Gentoo DVD and rsync to another ext3 partition, wipe the current filesystem and then rsync back. OK, I once again verified that fragmentation seems to be a big issue even on Linux. I just migrated to ext4, and in order to do that I had to rsync, format and rsync back. The result is similar to the last time I did this (over 8 months ago): emerge --sync takes 15 seconds (at least 3 minutes yesterday) update-eix takes 2 seconds (20 seconds yesterday) And I don't believe it's due to ext4. It's a nice speed-up from ext3, but not THAT nice. Well, try as I may, I could not get mine past 10% on resiserfs. Fragmentation happens on any file system but I think the point is that Linux doesn't get as bad as the windoze file system. 10% or so is not to bad depending on the size of the files. Files that are large will have to be fragmented no matter what file system you use. I posted in another the reply right after a copy to another drive. I think that was before I even booted into the OS and was still on the CD. It is around 2% or so. I doubt given that condition that it could get any better. Dale :-) :-)
[gentoo-user] Re: Fragmentation of my drives. Curious mostly
Dale wrote: Nikos Chantziaras wrote: OK, I once again verified that fragmentation seems to be a big issue even on Linux. I just migrated to ext4, and in order to do that I had to rsync, format and rsync back. The result is similar to the last time I did this (over 8 months ago): emerge --sync takes 15 seconds (at least 3 minutes yesterday) update-eix takes 2 seconds (20 seconds yesterday) And I don't believe it's due to ext4. It's a nice speed-up from ext3, but not THAT nice. Well, try as I may, I could not get mine past 10% on resiserfs. Fragmentation happens on any file system but I think the point is that Linux doesn't get as bad as the windoze file system. 10% or so is not to bad depending on the size of the files. Files that are large will have to be fragmented no matter what file system you use. I posted in another the reply right after a copy to another drive. I think that was before I even booted into the OS and was still on the CD. It is around 2% or so. I doubt given that condition that it could get any better. I think the main problem may not be so much fragmentation of files, but rather their position on disk. Even if files are not fragmented, if they are located too far from each other even though they're related (same directory for example) or there's simply too much empty space between files (I think this is intentional in order to reduce fragmentation) then seek times get really bad. After I rsync the data back, it's nicely and sequentially laid out on disk. I guess over time it starts to get further apart again (to combat fragmentation) and emerge --sync goes up from 15 seconds to 2 minutes again. Even though the files aren't fragmented at all. Some defrag apps for Windoze actually offer to put the files back closer together without trying to defragment at all. I guess this is why :P
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
On Friday 26 December 2008 21:49:02 Nikos Chantziaras wrote: OK, I once again verified that fragmentation seems to be a big issue even on Linux. I just migrated to ext4, and in order to do that I had to rsync, format and rsync back. The result is similar to the last time I did this (over 8 months ago): emerge --sync takes 15 seconds (at least 3 minutes yesterday) update-eix takes 2 seconds (20 seconds yesterday) And I don't believe it's due to ext4. It's a nice speed-up from ext3, but not THAT nice. Um, did it occur to you that after you emerge --sync'ed yesterday and ran update-eix that your portage tree is now very up to date and your eix cache is hot? Therefore successive runs will naturally be much quicker? And that yesterday was xmas day, a day most likely to involve very few if any portage updates? Or that emerge --sync could easily speed up simply because you had more bandwidth? Your speed-ups likely have very little to do with your filesystem. -- alan dot mckinnon at gmail dot com
[gentoo-user] Re: Fragmentation of my drives. Curious mostly
Alan McKinnon wrote: On Friday 26 December 2008 21:49:02 Nikos Chantziaras wrote: OK, I once again verified that fragmentation seems to be a big issue even on Linux. I just migrated to ext4, and in order to do that I had to rsync, format and rsync back. The result is similar to the last time I did this (over 8 months ago): emerge --sync takes 15 seconds (at least 3 minutes yesterday) update-eix takes 2 seconds (20 seconds yesterday) And I don't believe it's due to ext4. It's a nice speed-up from ext3, but not THAT nice. Um, did it occur to you that after you emerge --sync'ed yesterday and ran update-eix that your portage tree is now very up to date and your eix cache is hot? Therefore successive runs will naturally be much quicker? And that yesterday was xmas day, a day most likely to involve very few if any portage updates? Or that emerge --sync could easily speed up simply because you had more bandwidth? Your speed-ups likely have very little to do with your filesystem. Well, instead of yesterday let's just say the past 5 months. I already did the rsync/format thing a few times over the last years, and the results are always the same: very fast filesystem for about a month, then it starts getting slower over time.
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Nikos Chantziaras wrote: Dale wrote: Nikos Chantziaras wrote: OK, I once again verified that fragmentation seems to be a big issue even on Linux. I just migrated to ext4, and in order to do that I had to rsync, format and rsync back. The result is similar to the last time I did this (over 8 months ago): emerge --sync takes 15 seconds (at least 3 minutes yesterday) update-eix takes 2 seconds (20 seconds yesterday) And I don't believe it's due to ext4. It's a nice speed-up from ext3, but not THAT nice. Well, try as I may, I could not get mine past 10% on resiserfs. Fragmentation happens on any file system but I think the point is that Linux doesn't get as bad as the windoze file system. 10% or so is not to bad depending on the size of the files. Files that are large will have to be fragmented no matter what file system you use. I posted in another the reply right after a copy to another drive. I think that was before I even booted into the OS and was still on the CD. It is around 2% or so. I doubt given that condition that it could get any better. I think the main problem may not be so much fragmentation of files, but rather their position on disk. Even if files are not fragmented, if they are located too far from each other even though they're related (same directory for example) or there's simply too much empty space between files (I think this is intentional in order to reduce fragmentation) then seek times get really bad. After I rsync the data back, it's nicely and sequentially laid out on disk. I guess over time it starts to get further apart again (to combat fragmentation) and emerge --sync goes up from 15 seconds to 2 minutes again. Even though the files aren't fragmented at all. Some defrag apps for Windoze actually offer to put the files back closer together without trying to defragment at all. I guess this is why :P Well, this is what I got on my rig. Sort of interesting in a way. r...@smoker / # mount /dev/hda6 on / type reiserfs (rw) /proc on /proc type proc (rw) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec) udev on /dev type tmpfs (rw,nosuid) devpts on /dev/pts type devpts (rw,nosuid,noexec) /dev/hda1 on /boot type ext2 (rw) /dev/hda7 on /home type reiserfs (rw) /dev/hda8 on /usr/portage type ext2 (rw) /dev/hda9 on /data type reiserfs (rw) none on /dev/shm type tmpfs (rw) usbfs on /proc/bus/usb type usbfs (rw,noexec,nosuid,devmode=0664,devgid=85) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev) /dev/hdd on /media/hdd type udf (rw,noexec,nosuid,nodev,uid=0,gid=0,umask=007) r...@smoker / # r...@smoker / # /root/fragck.pl / 3.10978840211776% non contiguous files, 1.08156705459019 average fragments. r...@smoker / # /root/fragck.pl /usr/portage/ 0.0276657266269232% non contiguous files, 1.00029450612216 average fragments. r...@smoker / # /root/fragck.pl /boot/ 6.25% non contiguous files, 1.0625 average fragments. r...@smoker / # /root/fragck.pl /home/ 3.2440588457186% non contiguous files, 1.16408902301018 average fragments. r...@smoker / # /root/fragck.pl /data/ 5.56267766568196% non contiguous files, 1.06797837355777 average fragments. r...@smoker / # Now keep in mind that the first one includes all the others. I'm logged into a GUI so I can't umount /home at least. May do that in single mode someday. I think sometimes the files are just to big to fit on one section. I know I have some files that are pretty big. I got a couple videos that are big that came off my camera and one video that is a hour or so long. I think there are a lot of variables that without a microscope we can never see and know for sure. Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Nikos Chantziaras wrote: Alan McKinnon wrote: On Friday 26 December 2008 21:49:02 Nikos Chantziaras wrote: OK, I once again verified that fragmentation seems to be a big issue even on Linux. I just migrated to ext4, and in order to do that I had to rsync, format and rsync back. The result is similar to the last time I did this (over 8 months ago): emerge --sync takes 15 seconds (at least 3 minutes yesterday) update-eix takes 2 seconds (20 seconds yesterday) And I don't believe it's due to ext4. It's a nice speed-up from ext3, but not THAT nice. Um, did it occur to you that after you emerge --sync'ed yesterday and ran update-eix that your portage tree is now very up to date and your eix cache is hot? Therefore successive runs will naturally be much quicker? And that yesterday was xmas day, a day most likely to involve very few if any portage updates? Or that emerge --sync could easily speed up simply because you had more bandwidth? Your speed-ups likely have very little to do with your filesystem. Well, instead of yesterday let's just say the past 5 months. I already did the rsync/format thing a few times over the last years, and the results are always the same: very fast filesystem for about a month, then it starts getting slower over time. I have to say that after my recent transfer, my login got a whole one second faster. I can't tell any difference anywhere else. Of course, portage has always been on its own partition and used ext3. We need a hard drive engineer on here. :/ Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Dale wrote: I have to say that after my recent transfer, my login got a whole one second faster. I can't tell any difference anywhere else. Of course, portage has always been on its own partition and used ext3. We need a hard drive engineer on here. :/ Dale :-) :-) Hey, I've been following this thread with some interest. I just wanted to note that you guys might like to subscribe to Sun's ZFS-discuss list and possibly Storage-discuss. The guys on there really are hard-disk gurus and some of the things they talk about are miles over my head. It's just interesting as ZFS is supposedly (and I believe it) THE filesystem when it comes to combating fragmentation. Maybe reading over what those guys chat about would be interesting to some folks from this thread. In fact, the guys over at Sun are so hot on fighting fragmentation, they're already looking at some really advanced things like low level algorithms for deduplication and some other things that scare me and make me want to take a hot shower :P Happy holidays Matt
[gentoo-user] Re: Fragmentation of my drives. Curious mostly
Dale wrote: Nikos Chantziaras wrote: I already did the rsync/format thing a few times over the last years, and the results are always the same: very fast filesystem for about a month, then it starts getting slower over time. I have to say that after my recent transfer, my login got a whole one second faster. I can't tell any difference anywhere else. Of course, portage has always been on its own partition and used ext3. We need a hard drive engineer on here. :/ Well, I have everything in /. Except for /boot. Maybe I should reconsider my setup :P
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Matt Harrison wrote: Dale wrote: I have to say that after my recent transfer, my login got a whole one second faster. I can't tell any difference anywhere else. Of course, portage has always been on its own partition and used ext3. We need a hard drive engineer on here. :/ Dale :-) :-) Hey, I've been following this thread with some interest. I just wanted to note that you guys might like to subscribe to Sun's ZFS-discuss list and possibly Storage-discuss. The guys on there really are hard-disk gurus and some of the things they talk about are miles over my head. It's just interesting as ZFS is supposedly (and I believe it) THE filesystem when it comes to combating fragmentation. Maybe reading over what those guys chat about would be interesting to some folks from this thread. In fact, the guys over at Sun are so hot on fighting fragmentation, they're already looking at some really advanced things like low level algorithms for deduplication and some other things that scare me and make me want to take a hot shower :P Happy holidays Matt But if we learned to much, we may be dangerous or something. Sometimes to much knowledge can be bad. lol I !think! I tried XFS once. If it was XFS, you need to have a UPS for sure. Every time the system crashed I had to re-install. I never got it to recover even once. I have heard the same thing about its defrag efficiency tho. Just don't trust it to much with my data. Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Nikos Chantziaras wrote: Dale wrote: Nikos Chantziaras wrote: I already did the rsync/format thing a few times over the last years, and the results are always the same: very fast filesystem for about a month, then it starts getting slower over time. I have to say that after my recent transfer, my login got a whole one second faster. I can't tell any difference anywhere else. Of course, portage has always been on its own partition and used ext3. We need a hard drive engineer on here. :/ Well, I have everything in /. Except for /boot. Maybe I should reconsider my setup :P How you partition depends on what you are doing. I just have a desktop myself. I keep /home and /data separate in case I need to switch over for some reason and portage just to keep it from getting to fragmented. I guess it helps some at least. Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Dale wrote: But if we learned to much, we may be dangerous or something. Sometimes to much knowledge can be bad. lol I !think! I tried XFS once. If it was XFS, you need to have a UPS for sure. Every time the system crashed I had to re-install. I never got it to recover even once. I have heard the same thing about its defrag efficiency tho. Just don't trust it to much with my data. Dale :-) :-) Not sure if you're talking about something else but I was talking about ZFS[1], not XFS. ZFS is the latest filesystem from Sun which ships with the later versions of Solaris/OpenSolaris. I don't want to be seen to advertise it loads here, but it really is good. I recently moved my fileserver to a solaris/ZFS box instead of raid on gentoo. Since then my data hasn't been inaccessible once, and I haven't had the scary problems like when gentoo decides to reboot and not bring my arrays back online :) [1]http://en.wikipedia.org/wiki/ZFS Matt
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
On Fri, 26 Dec 2008 23:02:38 +0200, Nikos Chantziaras wrote: Well, instead of yesterday let's just say the past 5 months. I already did the rsync/format thing a few times over the last years, and the results are always the same: very fast filesystem for about a month, then it starts getting slower over time. I keep /usr/portage on a separate, small filesystem, using ext2. If it starts to slow down,I can simply reformat it and sync again. -- Neil Bothwick Some people are born mediocre, some people achieve mediocrity, and some people have mediocrity thrust upon them. - Joseph Heller, Catch-22 signature.asc Description: PGP signature
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
On Tuesday 16 December 2008, Miguel Ramos wrote: Another argument in favour of cp in Linux: holes in sparse files are kept correctly, whereas using tar they are not. It is curious that this is very OS dependent. In FreeBSD, with cp, holes always go away, using tar, or better dump/restore is a way to keep all file attributes. In Linux, cp -a seems to be better for archives than tar, because it preserves these properties better, even across devices. Hmm..., with tar, -p will preserve permissions and -S will handle sparce files efficiently. -W will additionally verify that that data was archived without corruption. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
On Tue, 16 Dec 2008 19:06:06 -0600, Dale wrote: Light bulb warning. So null and console are on the drive for it to start up but once it mounts /dev then it uses that virtual thing? Cool, if I understand that correctly. Yes, those two devices are needed before udev starts,so they have to be on the root filesystem. If you have anything else in dev on the root filesystem, you are only wasting space. -- Neil Bothwick WinErr 007: System price error - Inadequate money spent on hardware signature.asc Description: PGP signature
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
On Wed, 17 Dec 2008 06:20:38 -0600, Dale wrote: I got it transfered over. I noticed something weird tho. I was booted from the CD. When I was checking the permissions to make sure things were going well, it kept showing gentoo:users instead of dale:users for example. The ones that were root were fine but the ones that should be dale:users was gentoo:users. I stopped and reformatted the drives and it always did the same thing. I finally gave in and let it copy anyway. After it was copied, I chroot'ed in and all the permissions were like they should be including dale:users. Any idea why it did that? It did the same thing with both rsync -ax and cp -av. Just thought it was weird is all. Filesystems store numeric values for UID/GID, commands like ls translate these to actual names. Gentoo normally makes the first user 1000, which is probably the UID of dale on your installation and gentoo on the live CD.Root is always UID 0, which is why that was shown correctly. -- Neil Bothwick I doubt therefore I might be. signature.asc Description: PGP signature
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Neil Bothwick wrote: On Wed, 17 Dec 2008 06:20:38 -0600, Dale wrote: I got it transfered over. I noticed something weird tho. I was booted from the CD. When I was checking the permissions to make sure things were going well, it kept showing gentoo:users instead of dale:users for example. The ones that were root were fine but the ones that should be dale:users was gentoo:users. I stopped and reformatted the drives and it always did the same thing. I finally gave in and let it copy anyway. After it was copied, I chroot'ed in and all the permissions were like they should be including dale:users. Any idea why it did that? It did the same thing with both rsync -ax and cp -av. Just thought it was weird is all. Filesystems store numeric values for UID/GID, commands like ls translate these to actual names. Gentoo normally makes the first user 1000, which is probably the UID of dale on your installation and gentoo on the live CD.Root is always UID 0, which is why that was shown correctly. Oh, makes sense. Should have known that computers reduce everything to numbers. ROFLMAO At least now I know why it did that. It had me freaked out for a bit there. New transfer is working very well. Pretty swift but not much difference from the old one. At least I got some of the cruft cleaned out. Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Dale wrote: Dale wrote: This is interesting. I am starting a new install on my backup drive. I'm part way through the install, fetching all the KDE stuff right now. This is what I got from the little frag script: r...@smoker / # /root/fragck.pl /backup/ 0.953336175120985% non contiguous files, 1.02414182192021 average fragments. r...@smoker / # Less than 1% is my starting point I guess. This currently has ext3 on it. I did start out with a freshly formatted file system. Also, this is all on one big partition at the moment. I'll post later what it says after compiling a few packages. I figure KDE should stir up something. LOL Dale :-) :-) This is after a almost complete install. About to start OOo. r...@smoker / # /root/fragck.pl /backup/ 2.00854614717917% non contiguous files, 1.04611358582092 average fragments. r...@smoker / # r...@smoker / # du -shc /backup/ 5.6G/backup/ 5.6Gtotal r...@smoker / # I would assume that would be something like it was when I started my current install years ago. Which is at 10% or so now. Thoughts anyone? Dale :-) :-) OK. I completed my install and got everything working. This is what I got after that: 2.24954051453251% non contiguous files, 1.06439409487064 average fragments. I then ran shake just to see if it changed for the better or worse. I got this surprising answer: 25.2668178520421% non contiguous files, 1.41060290111655 average fragments. You may want to look twice at the decimal point. It appears that shake makes things much worse or the fragck script has some serious issues one. I have no clue which. I'm not to worried about this since I will be moving this over to the other drive anyway. I would like to know what command I should use to tar up everything, transfer it over and untar it all on one line if possible? I plan to do this while booted from a Gentoo CD. I just want to try this so that it will be compressed then transfered and untared once on the way. Does this make since? I have used cp -av in the past. Thanks. Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Am Dienstag, den 16.12.2008, 01:59 -0600 schrieb Dale: Dale wrote: Dale wrote: This is interesting. I am starting a new install on my backup drive. I'm part way through the install, fetching all the KDE stuff right now. This is what I got from the little frag script: r...@smoker / # /root/fragck.pl /backup/ 0.953336175120985% non contiguous files, 1.02414182192021 average fragments. r...@smoker / # Less than 1% is my starting point I guess. This currently has ext3 on it. I did start out with a freshly formatted file system. Also, this is all on one big partition at the moment. I'll post later what it says after compiling a few packages. I figure KDE should stir up something. LOL Dale :-) :-) This is after a almost complete install. About to start OOo. r...@smoker / # /root/fragck.pl /backup/ 2.00854614717917% non contiguous files, 1.04611358582092 average fragments. r...@smoker / # r...@smoker / # du -shc /backup/ 5.6G/backup/ 5.6Gtotal r...@smoker / # I would assume that would be something like it was when I started my current install years ago. Which is at 10% or so now. Thoughts anyone? Dale :-) :-) OK. I completed my install and got everything working. This is what I got after that: 2.24954051453251% non contiguous files, 1.06439409487064 average fragments. I then ran shake just to see if it changed for the better or worse. I got this surprising answer: 25.2668178520421% non contiguous files, 1.41060290111655 average fragments. You may want to look twice at the decimal point. It appears that shake makes things much worse or the fragck script has some serious issues one. I have no clue which. I'm not to worried about this since I will be moving this over to the other drive anyway. I would like to know what command I should use to tar up everything, transfer it over and untar it all on one line if possible? I plan to do this while booted from a Gentoo CD. I just want to try this so that it will be compressed then transfered and untared once on the way. Does this make since? I have used cp -av in the past. Thanks. Dale :-) :-) With transfer do you mean over a network, or to another local drive? You can of course use something like # tar czpf - | ssh remote - tar xzpf -C /dir (above probably not syntactically correct), but there are faster and easier options: cp -a costs little resources locally and maintains POSIX permissions, while rsync -aASH --numeric-ids is perfect for remote copy. You can use rsync also locally. It will (with the -A switch) also transfer POSIX-ACLs, if that is of any concern. It is also useful, if a transfer breaks at some moment, because it will kind of continue it :) Omiting the -v switch can significantly speed up things - depends on your terminal. In every case it helps to only see the errors, and not let them scroll away by everything that went well. Bye, Daniel -- PGP key: http://pgpkeys.pca.dfn.de/pks/lookup?search=0xBB9D4887op=get # gpg --recv-keys --keyserver hkp://subkeys.pgp.net 0xBB9D4887 signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Daniel Troeder wrote: Am Dienstag, den 16.12.2008, 01:59 -0600 schrieb Dale: I'm not to worried about this since I will be moving this over to the other drive anyway. I would like to know what command I should use to tar up everything, transfer it over and untar it all on one line if possible? I plan to do this while booted from a Gentoo CD. I just want to try this so that it will be compressed then transfered and untared once on the way. Does this make since? I have used cp -av in the past. Thanks. Dale :-) :-) With transfer do you mean over a network, or to another local drive? You can of course use something like # tar czpf - | ssh remote - tar xzpf -C /dir (above probably not syntactically correct), but there are faster and easier options: cp -a costs little resources locally and maintains POSIX permissions, while rsync -aASH --numeric-ids is perfect for remote copy. You can use rsync also locally. It will (with the -A switch) also transfer POSIX-ACLs, if that is of any concern. It is also useful, if a transfer breaks at some moment, because it will kind of continue it :) Omiting the -v switch can significantly speed up things - depends on your terminal. In every case it helps to only see the errors, and not let them scroll away by everything that went well. Bye, Daniel The drive is in the same machine so there is no network involved. Should help make it a little more simple. Would this work? tar czpf - | tar xzpf -C /dir Basically, I want as clean a file system as I can get to start off with at least. Goal is very little fragmentation. Thanks Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Am Dienstag, den 16.12.2008, 03:15 -0600 schrieb Dale: Daniel Troeder wrote: Am Dienstag, den 16.12.2008, 01:59 -0600 schrieb Dale: I'm not to worried about this since I will be moving this over to the other drive anyway. I would like to know what command I should use to tar up everything, transfer it over and untar it all on one line if possible? I plan to do this while booted from a Gentoo CD. I just want to try this so that it will be compressed then transfered and untared once on the way. Does this make since? I have used cp -av in the past. Thanks. Dale :-) :-) With transfer do you mean over a network, or to another local drive? You can of course use something like # tar czpf - | ssh remote - tar xzpf -C /dir (above probably not syntactically correct), but there are faster and easier options: cp -a costs little resources locally and maintains POSIX permissions, while rsync -aASH --numeric-ids is perfect for remote copy. You can use rsync also locally. It will (with the -A switch) also transfer POSIX-ACLs, if that is of any concern. It is also useful, if a transfer breaks at some moment, because it will kind of continue it :) Omiting the -v switch can significantly speed up things - depends on your terminal. In every case it helps to only see the errors, and not let them scroll away by everything that went well. Bye, Daniel The drive is in the same machine so there is no network involved. Should help make it a little more simple. Would this work? tar czpf - | tar xzpf -C /dir Basically, I want as clean a file system as I can get to start off with at least. Goal is very little fragmentation. Thanks Dale While this will work perfectly well, this command is a waste of resources. The compression (-z) makes locally no sense, and there is no need to tar the data (which will basically just concat files). You will get the exact same result with # cp -a /source /dest If the FS has been formatted before, no fragmentation should occur in every scenario, as long as no parallelism is used while copying, because each file will be created and filled with data one after another. Bye, Daniel signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Another argument in favour of cp in Linux: holes in sparse files are kept correctly, whereas using tar they are not. It is curious that this is very OS dependent. In FreeBSD, with cp, holes always go away, using tar, or better dump/restore is a way to keep all file attributes. In Linux, cp -a seems to be better for archives than tar, because it preserves these properties better, even across devices. 2008/12/16 Dale rdalek1...@gmail.com: Daniel Troeder wrote: Am Dienstag, den 16.12.2008, 03:15 -0600 schrieb Dale: [...] While this will work perfectly well, this command is a waste of resources. The compression (-z) makes locally no sense, and there is no need to tar the data (which will basically just concat files). You will get the exact same result with # cp -a /source /dest If the FS has been formatted before, no fragmentation should occur in every scenario, as long as no parallelism is used while copying, because each file will be created and filled with data one after another. Bye, Daniel Cool. Then I can just use cp -a and let her rip. I plan to redo my partitions so I will have to reformat the partitions too. I guess this will be as good as it gets. I'll also report the results of fragck when I get this done. Just curious myself. I think I will skip shake this time tho. ;-) Thanks much. Dale -- Miguel Ramos 2...@miguel.ramos.name GnuPG ID 0xA006A14C
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
On Tue, 16 Dec 2008 10:32:00 +0100, Daniel Troeder wrote: While this will work perfectly well, this command is a waste of resources. The compression (-z) makes locally no sense, and there is no need to tar the data (which will basically just concat files). You will get the exact same result with # cp -a /source /dest There is one slight disadvantage to cp in that it changed the modified time of directories to the current time, which rsync does not. I'd use rsync -ax /source/ /dest/ -- Neil Bothwick If ignorance is bliss, you must be orgasmic. signature.asc Description: PGP signature
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Daniel Troeder wrote: Am Dienstag, den 16.12.2008, 03:15 -0600 schrieb Dale: Daniel Troeder wrote: Am Dienstag, den 16.12.2008, 01:59 -0600 schrieb Dale: I'm not to worried about this since I will be moving this over to the other drive anyway. I would like to know what command I should use to tar up everything, transfer it over and untar it all on one line if possible? I plan to do this while booted from a Gentoo CD. I just want to try this so that it will be compressed then transfered and untared once on the way. Does this make since? I have used cp -av in the past. Thanks. Dale :-) :-) With transfer do you mean over a network, or to another local drive? You can of course use something like # tar czpf - | ssh remote - tar xzpf -C /dir (above probably not syntactically correct), but there are faster and easier options: cp -a costs little resources locally and maintains POSIX permissions, while rsync -aASH --numeric-ids is perfect for remote copy. You can use rsync also locally. It will (with the -A switch) also transfer POSIX-ACLs, if that is of any concern. It is also useful, if a transfer breaks at some moment, because it will kind of continue it :) Omiting the -v switch can significantly speed up things - depends on your terminal. In every case it helps to only see the errors, and not let them scroll away by everything that went well. Bye, Daniel The drive is in the same machine so there is no network involved. Should help make it a little more simple. Would this work? tar czpf - | tar xzpf -C /dir Basically, I want as clean a file system as I can get to start off with at least. Goal is very little fragmentation. Thanks Dale While this will work perfectly well, this command is a waste of resources. The compression (-z) makes locally no sense, and there is no need to tar the data (which will basically just concat files). You will get the exact same result with # cp -a /source /dest If the FS has been formatted before, no fragmentation should occur in every scenario, as long as no parallelism is used while copying, because each file will be created and filled with data one after another. Bye, Daniel Cool. Then I can just use cp -a and let her rip. I plan to redo my partitions so I will have to reformat the partitions too. I guess this will be as good as it gets. I'll also report the results of fragck when I get this done. Just curious myself. I think I will skip shake this time tho. ;-) Thanks much. Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Neil Bothwick wrote: On Tue, 16 Dec 2008 10:32:00 +0100, Daniel Troeder wrote: While this will work perfectly well, this command is a waste of resources. The compression (-z) makes locally no sense, and there is no need to tar the data (which will basically just concat files). You will get the exact same result with # cp -a /source /dest There is one slight disadvantage to cp in that it changed the modified time of directories to the current time, which rsync does not. I'd use rsync -ax /source/ /dest/ I made a note of that command and will give that a try. I'll also read the man page to see how to get it to skip /dev /sys /proc etc etc. Thanks Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
On Tue, 16 Dec 2008 15:34:28 -0600, Dale wrote: rsync -ax /source/ /dest/ I made a note of that command and will give that a try. I'll also read the man page to see how to get it to skip /dev /sys /proc etc etc. That's what the -x is for. -- Neil Bothwick Be nice to moderators. They HATE that! signature.asc Description: PGP signature
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Neil Bothwick wrote: On Tue, 16 Dec 2008 15:34:28 -0600, Dale wrote: rsync -ax /source/ /dest/ I made a note of that command and will give that a try. I'll also read the man page to see how to get it to skip /dev /sys /proc etc etc. That's what the -x is for. Thanks for the info. As you may can tell, I have never used rsync before. :/ Then again, since I will be booted from the CD, shouldn't they be empty anyway? Except maybe for /dev/null and /dev/console I guess? I got to go visit a friend so it may be tomorrow before I get to transfer now. Thanks again. Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
On Tue, 16 Dec 2008 17:49:11 -0600, Dale wrote: I made a note of that command and will give that a try. I'll also read the man page to see how to get it to skip /dev /sys /proc etc etc. That's what the -x is for. Thanks for the info. As you may can tell, I have never used rsync before. :/ Your portage tree must be very outdated by now :P Then again, since I will be booted from the CD, shouldn't they be empty anyway? Except maybe for /dev/null and /dev/console I guess? Yes, and that is a better way of doing it as you will copy the two files in the underlying /dev. -x tells rsync to not cross filesystem boundaries, so it will make no difference if you are working from a live CD. -- Neil Bothwick Exercise daily. Eat wisely. Die anyway. signature.asc Description: PGP signature
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Neil Bothwick wrote: On Tue, 16 Dec 2008 17:49:11 -0600, Dale wrote: I made a note of that command and will give that a try. I'll also read the man page to see how to get it to skip /dev /sys /proc etc etc. That's what the -x is for. Thanks for the info. As you may can tell, I have never used rsync before. :/ Your portage tree must be very outdated by now :P Then again, since I will be booted from the CD, shouldn't they be empty anyway? Except maybe for /dev/null and /dev/console I guess? Yes, and that is a better way of doing it as you will copy the two files in the underlying /dev. -x tells rsync to not cross filesystem boundaries, so it will make no difference if you are working from a live CD. Well, I have used eix-sync but not rsync directly as a command. I sync once a week, give or take. Light bulb warning. So null and console are on the drive for it to start up but once it mounts /dev then it uses that virtual thing? Cool, if I understand that correctly. I like it when my light bulb comes on. It's so . . . . . pretty. LOL Dale :-) :-) P. S. My friend got called in to work. May be done tonight after all. :-(
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Dale wrote: This is interesting. I am starting a new install on my backup drive. I'm part way through the install, fetching all the KDE stuff right now. This is what I got from the little frag script: [EMAIL PROTECTED] / # /root/fragck.pl /backup/ 0.953336175120985% non contiguous files, 1.02414182192021 average fragments. [EMAIL PROTECTED] / # Less than 1% is my starting point I guess. This currently has ext3 on it. I did start out with a freshly formatted file system. Also, this is all on one big partition at the moment. I'll post later what it says after compiling a few packages. I figure KDE should stir up something. LOL Dale :-) :-) This is after a almost complete install. About to start OOo. [EMAIL PROTECTED] / # /root/fragck.pl /backup/ 2.00854614717917% non contiguous files, 1.04611358582092 average fragments. [EMAIL PROTECTED] / # [EMAIL PROTECTED] / # du -shc /backup/ 5.6G/backup/ 5.6Gtotal [EMAIL PROTECTED] / # I would assume that would be something like it was when I started my current install years ago. Which is at 10% or so now. Thoughts anyone? Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Alan McKinnon wrote: Only a proper analysis of your files will tell you this. It's easy enough to check for individual file fragmentation and get stats on that before you do the copy-off/copy-back. This is interesting. I am starting a new install on my backup drive. I'm part way through the install, fetching all the KDE stuff right now. This is what I got from the little frag script: [EMAIL PROTECTED] / # /root/fragck.pl /backup/ 0.953336175120985% non contiguous files, 1.02414182192021 average fragments. [EMAIL PROTECTED] / # Less than 1% is my starting point I guess. This currently has ext3 on it. I did start out with a freshly formatted file system. Also, this is all on one big partition at the moment. I'll post later what it says after compiling a few packages. I figure KDE should stir up something. LOL Dale :-) :-)
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
On Friday 28 November 2008 20:24:38 Nikos Chantziaras wrote: Alan McKinnon wrote: On Friday 28 November 2008 13:14:42 Dale wrote: If this is a little high, what would be the best way to defrag it? By not defragging it. It's not Windows. Windows boxes needs defragging not because fragmentation is a huge problem in itself, but because windows filesystems are a steaming mess of [EMAIL PROTECTED] that do little right and most things wrong. Defrag treats the symptom, not the cause :-) I don't buy into that argument and never did. Every few months I copy the whole HD to another one and then back to counter fragmentation (ext3) and the system becomes noticeably faster after doing it (speed increase in emerge --sync for example.) Maybe it's not fragmentation but rather related files being more closely together after I do this. Only a proper analysis of your files will tell you this. It's easy enough to check for individual file fragmentation and get stats on that before you do the copy-off/copy-back. -- alan dot mckinnon at gmail dot com
[gentoo-user] Re: Fragmentation of my drives. Curious mostly
Alan McKinnon wrote: On Friday 28 November 2008 13:14:42 Dale wrote: If this is a little high, what would be the best way to defrag it? By not defragging it. It's not Windows. Windows boxes needs defragging not because fragmentation is a huge problem in itself, but because windows filesystems are a steaming mess of [EMAIL PROTECTED] that do little right and most things wrong. Defrag treats the symptom, not the cause :-) I don't buy into that argument and never did. Every few months I copy the whole HD to another one and then back to counter fragmentation (ext3) and the system becomes noticeably faster after doing it (speed increase in emerge --sync for example.) Maybe it's not fragmentation but rather related files being more closely together after I do this.
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
[...] what would be the best way to defrag it? By not defragging it. It's not Windows. Windows boxes needs defragging not because fragmentation is a huge problem in itself, but because windows filesystems are a steaming mess of [EMAIL PROTECTED] that do little right and most things wrong. Defrag treats the symptom, not the cause :-) I don't buy into that argument and never did. Every few months I copy the whole HD to another one and then back to counter fragmentation (ext3) and the system becomes noticeably faster after doing it (speed increase in emerge --sync for example.) Maybe it's not fragmentation but rather related files being more closely together after I do this. How exactly do you copy the files? Be careful not to lose some file property. How about sparse files, for example? AFAIK, you can make a complete backup of a filesytem with (as root, running from another system - such as a liveCD) $ cd /path/to/mountpoint $ tar -cSv -f /path/to/tarball.tar . But I am not sure.
[gentoo-user] Re: Fragmentation of my drives. Curious mostly
Jorge Peixoto de Morais Neto wrote: [...] what would be the best way to defrag it? By not defragging it. It's not Windows. Windows boxes needs defragging not because fragmentation is a huge problem in itself, but because windows filesystems are a steaming mess of [EMAIL PROTECTED] that do little right and most things wrong. Defrag treats the symptom, not the cause :-) I don't buy into that argument and never did. Every few months I copy the whole HD to another one and then back to counter fragmentation (ext3) and the system becomes noticeably faster after doing it (speed increase in emerge --sync for example.) Maybe it's not fragmentation but rather related files being more closely together after I do this. How exactly do you copy the files? Be careful not to lose some file property. How about sparse files, for example? AFAIK, you can make a complete backup of a filesytem with (as root, running from another system - such as a liveCD) $ cd /path/to/mountpoint $ tar -cSv -f /path/to/tarball.tar . But I am not sure. I simply boot from the Gentoo DVD and rsync to another ext3 partition, wipe the current filesystem and then rsync back.
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
On 28 Nov 2008, at 19:27, Jorge Peixoto de Morais Neto wrote: ... I don't buy into that argument and never did. Every few months I copy the whole HD to another one and then back to counter fragmentation (ext3) and the system becomes noticeably faster after doing it (speed increase in emerge --sync for example.) Maybe it's not fragmentation but rather related files being more closely together after I do this. How exactly do you copy the files? Be careful not to lose some file property. How about sparse files, for example? AFAIK, you can make a complete backup of a filesytem with (as root, running from another system - such as a liveCD) $ cd /path/to/mountpoint $ tar -cSv -f /path/to/tarball.tar . Shouldn't creating a stage4 be safe for this? There are a number of scripts on the forums (or the wiki?) that are supposed to be safe for creating stage4s from a live system. I think this would result in minimal downtime.(??) Stroller.
Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly
Jorge Peixoto de Morais Neto wrote: [...] what would be the best way to defrag it? By not defragging it. It's not Windows. Windows boxes needs defragging not because fragmentation is a huge problem in itself, but because windows filesystems are a steaming mess of [EMAIL PROTECTED] that do little right and most things wrong. Defrag treats the symptom, not the cause :-) I don't buy into that argument and never did. Every few months I copy the whole HD to another one and then back to counter fragmentation (ext3) and the system becomes noticeably faster after doing it (speed increase in emerge --sync for example.) Maybe it's not fragmentation but rather related files being more closely together after I do this. How exactly do you copy the files? Be careful not to lose some file property. How about sparse files, for example? AFAIK, you can make a complete backup of a filesytem with (as root, running from another system - such as a liveCD) $ cd /path/to/mountpoint $ tar -cSv -f /path/to/tarball.tar . But I am not sure. I use cp -av to copy mine. From what I have read it keeps permission, links and everything. I have done it before and it worked fine but that was before udev came along. Also, I do that booted from a CD, either Knoppix or Gentoo CD. I would think that since everything is wiped out in /dev/ when I shutdown, at least that is how it is set anyway, that udev has to recreate all the files in /dev/ during boot up. Of course, I have never checked that to make sure. Dale :-) :-)
