Re: [reiserfs-list] Silly question, defrag
On Wed, Apr 03, 2002 at 08:08:21AM -0800, Matthew Johnson wrote: On Wednesday 03 April 2002 00:21, Joe Cooper wrote: Don't Well I don't, but when newbies who are used to computing on win32 systems hear that they may not just accept the word don't. Actually its hard to find the reasons exactly why one does not defrag. It is more useful to look at why one DID defrag back in the bad ol' days of DOS and Windows. IIRC, the FAT filesystem would scan through it's equivalient of the free block list and start writing at the first free block. If it wrote for a while and then there was other data in the way it stop and go to the next free space. This way fragmentation was practically guarenteed and it happened rapidly. Modern filesystems use much smarter ways of laying out data on the disk so that fragmentation happens much less often. Now you will almost certainly waste more time by defragmenting than you would suffering whatever performance hit the little fragmentation there is causes. I've been using Linux/Unix for 10 years and I have never (not once!) defragged a filesystem. Perhaps I should aim this message to the kernel mailing list, so that I can get response from a wider array of people who like other filesystems. But its not kernel related. I wouldn't recommend doing that. The answer is pretty much the same regardless of the filesystem. If it's a non-FAT fs you probably don't have to worry about fragmentation. I do not agree. I run a fileserver with a 814GB filesystem using ReiserFS (I have run NTFS and ext2/3 also). Modern filesystems might be smarter in storing new files by not packing them tightly. In my case that workes fine up to a certain percentage, after that ALL new files are beeing fragmented due to the fact that there is only small blocks of space between all files. I don't see any filesystem that don't need defragmentation. Not in my case. //Anders
Re: [reiserfs-list] Silly question, defrag
On Thu, Apr 04, 2002 at 11:16:51AM +0200, Anders Widman wrote: I do not agree. I run a fileserver with a 814GB filesystem using ReiserFS (I have run NTFS and ext2/3 also). Modern filesystems might be smarter in storing new files by not packing them tightly. In my case that workes fine up to a certain percentage, after that ALL new files are beeing fragmented due to the fact that there is only small blocks of space between all files. I don't see any filesystem that don't need defragmentation. Not in my case. Yes, after a certain percentage you will start getting fragmentation. Will you really notice the performance hit? Who knows. It depends on how much and which files get fragmented. One solution is to not fill disks up to more than 90%. That is what most people who have a need to worry about such things do. I'm just saying that it isn't worth it. If you are at 90% you need to buy more disk anyway because soon you will be at 100%. If there were real value in regularly defragging then Veritas, Sun, IBM, HP, and all of those guys would have made defraggers for their respective filesystems and it would be considered best practice and standard operating procedure to use them. But I have never heard of any such tools nor procedures. This whole discussion results from the fact that so many Linux people come from PC backgrounds where they were taught to habitually defrag. People who come from other systems never give it a thought. Nonetheless, I look forward to having this functionality is reiserfs because it certainly can't hurt. What interests me even more than defragging is performance optimized layouts. If the filesystem can somehow keep track of patterns of frequently accessed blocks and could recognize that one set of blocks on the inner cylinders is always read immediately after reading a set of blocks on the outer cylinder (or perhaps instead of keeping track of blocks which are read it would be more efficient to keep track of commonly performed long seeks and move data to remove those) and could rearrange things so that all of the needed data passes under the read head in most often used sequence we would see a MUCH bigger improvement in performance. -- Tracy Reed http://www.ultraviolet.org Would you buy a car with the hood welded shut? Linux: the maintainable OS. msg05017/pgp0.pgp Description: PGP signature
Re: [reiserfs-list] Silly question, defrag
On Thu, 04 Apr 2002 10:54:40 PST, Tracy R Reed [EMAIL PROTECTED] said: If there were real value in regularly defragging then Veritas, Sun, IBM, HP, and all of those guys would have made defraggers for their respective filesystems and it would be considered best practice and standard operating procedure to use them. But I have never heard of any such tools nor procedures. Actually, IBM *does* ship a 'defragfs' with its JFS file system under AIX. When using file system compression under JFS, you can get horribly fragmented under certain usage patterns. 'defragfs' is perfectly happy with running on a live mounted filesystem. -- Valdis Kletnieks Computer Systems Senior Engineer Virginia Tech msg05018/pgp0.pgp Description: PGP signature
Re: [reiserfs-list] Silly question, defrag
On Thu, Apr 04, 2002 at 11:16:51AM +0200, Anders Widman wrote: I do not agree. I run a fileserver with a 814GB filesystem using ReiserFS (I have run NTFS and ext2/3 also). Modern filesystems might be smarter in storing new files by not packing them tightly. In my case that workes fine up to a certain percentage, after that ALL new files are beeing fragmented due to the fact that there is only small blocks of space between all files. I don't see any filesystem that don't need defragmentation. Not in my case. Yes, after a certain percentage you will start getting fragmentation. Will you really notice the performance hit? Who knows. It depends on how much and which files get fragmented. One solution is to not fill disks up to more than 90%. That is what most people who have a need to worry about such things do. I'm just saying that it isn't worth it. If you are at 90% you need to buy more disk anyway because soon you will be at 100%. Yes, The perfomance hit is very noticeable. Also, consiber a system with high uptime and many reads an writes (like my fileserver). Even though a filsystem is better than other it only takes a little longer before you see big framentation. If there were real value in regularly defragging then Veritas, Sun, IBM, HP, and all of those guys would have made defraggers for their respective filesystems and it would be considered best practice and standard operating procedure to use them. But I have never heard of any such tools nor procedures. Isn't Microsoft considered a large company? Actually, Microsoft said/stated that NTFS never needed defragmentation when Windows NT came. It (Microsoft) was still holding on to this 'fact' with Windows NT 4.0. The company, however, did change its policy with Windows 2000 and Windows XP. This whole discussion results from the fact that so many Linux people come from PC backgrounds where they were taught to habitually defrag. People who come from other systems never give it a thought. Are you sure the opposite is not true as well? It would seem that they (people from 'other' systems) are also thought defrag is only for windows. Nonetheless, I look forward to having this functionality is reiserfs because it certainly can't hurt. What interests me even more than defragging is performance optimized layouts. If the filesystem can somehow keep track of patterns of frequently accessed blocks and could recognize that one set of blocks on the inner cylinders is always read immediately after reading a set of blocks on the outer cylinder (or perhaps instead of keeping track of blocks which are read it would be more efficient to keep track of commonly performed long seeks and move data to remove those) and could rearrange things so that all of the needed data passes under the read head in most often used sequence we would see a MUCH bigger improvement in performance. I agree. Preventing (minimizing) fragmentation is probably the best choice, if not affecting write performance. //Anders
Re: [reiserfs-list] Silly question, defrag
Tracy R Reed wrote: If there were real value in regularly defragging then Veritas, Sun, IBM, HP, and all of those guys would have made defraggers for their respective filesystems and it would be considered best practice and standard operating procedure to use them. But I have never heard of any such tools nor procedures. HP-UX's vxfs (veritas i believe) ships with a defrag tool that packs extents and reorders data based on access usage (dir order I believe). HPUX documentation did recommend running it and tuning how often by ammount of fragmentation. I found on a heavily used ClearCase server a nightly run kept fragmentation down; although a weekly run was probably all that was necessary. Haven't run HPUX in several years so things may be different now. rbz -- Rodd Zurcher Principal Software Engineer SPS/WBSG/WSAS - 847.576.0666 [EMAIL PROTECTED]
Re: [reiserfs-list] Silly question, defrag
Defragging is a missing feature. It will be the most important feature of V4.1 4.1 seems pretty exciting, but why the need to defrag on these filesystems vs others. Or do we not yet have the data? Guess I am intrigued on how these work. Will defrag be included for peace of mind, or for real gain? How much fragmentation can one expect? Will you need to unmount drives to defrag? Sorry for all the questions... Matt
Re: [reiserfs-list] Silly question, defrag
Matthew Johnson wrote: On Wednesday 03 April 2002 00:21, Joe Cooper wrote: Don't Well I don't, but when newbies who are used to computing on win32 systems hear that they may not just accept the word don't. Actually its hard to find the reasons exactly why one does not defrag. ;-) ReiserFS (and ext2|3) do fragment somewhat, but the impact is not worth fighting over on most systems (certain environments are impacted more than others--mail servers and web caches being two examples that are hit pretty hard by fragmentation performance degradation). Besides, there is no method to defrag ReiserFS that I know of. Hans plans repacking in some future version. It will be nice, but the whole 'defrag once a month to keep your computer running smoothly' is kind of a Windows thing. Us Unix users don't really need to think so much on those sorts of things. Perhaps I should aim this message to the kernel mailing list, so that I can get response from a wider array of people who like other filesystems. But its not kernel related. All this because of a simple query someone posted to a user group mailing list lol. Matt There are ways in which Windows is better than Linux. This is one. To think that Linux is better for not having defrag is wishful innacurate thinking. Such is life. Send us $30k and defrag will go into v4.0 instead of 4.1.:) Hans
Re: [reiserfs-list] Silly question, defrag
Matthew Johnson wrote: Defragging is a missing feature. It will be the most important feature of V4.1 4.1 seems pretty exciting, but why the need to defrag on these filesystems vs others. Or do we not yet have the data? Guess I am intrigued on how these work. Will defrag be included for peace of mind, or for real gain? How much fragmentation can one expect? Will you need to unmount drives to defrag? Sorry for all the questions... Matt The reiser4.1 repacker will not just defrag, it will repack, and maybe even someday compress unaccessed data. There are a whole host of optimizations that are too expensive to do with every write, but very reasonable to do once a day. Repack will be great! As for why is it not in V3, the reason is simple. Code freeze hit, and it was not yet started Hans
Re: [reiserfs-list] Silly question, defrag
Don't ;-) ReiserFS (and ext2|3) do fragment somewhat, but the impact is not worth fighting over on most systems (certain environments are impacted more than others--mail servers and web caches being two examples that are hit pretty hard by fragmentation performance degradation). Besides, there is no method to defrag ReiserFS that I know of. Hans plans repacking in some future version. It will be nice, but the whole 'defrag once a month to keep your computer running smoothly' is kind of a Windows thing. Us Unix users don't really need to think so much on those sorts of things. Fragmentation is a problem with all filesystems. There is generally no way around fragmentation other than defragment. If you want to add/store a large file on a 30% full filesystem it would probably be stored on the first contingous area of free space. This works fine until you have used most of the space and changed the sizes of lots of files. Fragmentation is enevitable when you only have small contingous blocks of free/unallocated space and want to add a larger file. After some time you end up with heavily fragmentation on any filesystem. Of course, this doesn't happen when you don't add or change files. //Anders Matthew Johnson wrote: This is kind of a general silly question, but one that crops now and again. Especially from newbies... Whats the best, most accurate answer to give to a newbie when they ask how to defrag their hard drive, and does ReiserFS vary in itself with regards to this, with say ext2? Its just a question I sometimes get and wondered the best answer to this. Kind regards, Matt
Re: [reiserfs-list] Silly question, defrag
Anders Widman wrote: Don't ;-) ReiserFS (and ext2|3) do fragment somewhat, but the impact is not worth fighting over on most systems (certain environments are impacted more than others--mail servers and web caches being two examples that are hit pretty hard by fragmentation performance degradation). Besides, there is no method to defrag ReiserFS that I know of. Hans plans repacking in some future version. It will be nice, but the whole 'defrag once a month to keep your computer running smoothly' is kind of a Windows thing. Us Unix users don't really need to think so much on those sorts of things. Fragmentation is a problem with all filesystems. There is generally no way around fragmentation other than defragment. If you want to add/store a large file on a 30% full filesystem it would probably be stored on the first contingous area of free space. This works fine until you have used most of the space and changed the sizes of lots of files. Fragmentation is enevitable when you only have small contingous blocks of free/unallocated space and want to add a larger file. After some time you end up with heavily fragmentation on any filesystem. Of course, this doesn't happen when you don't add or change files. //Anders This is one of things (fragmentation) about which people say well known problem. Can you offer some solution? -- Yury Umanets IT Engineer of Priocom Corp. Phone: +380 44 2011959, ICQ: 55494590
Re: [reiserfs-list] Silly question, defrag
There are ways in which Windows is better than Linux. This is one. To think that Linux is better for not having defrag is wishful innacurate thinking. Such is life. Send us $30k and defrag will go into v4.0 instead of 4.1.:) Thanks for the answers, obviously 4.1 will be good thing. As for the cash, well *cough* I can defiantely wait :). Although I imagine that filesystems fragment differently depending on a number of factors (and that others are more prone to more fragmentation...). Does ReiserFS deal with fragmentation in any special way to minimise (ack, in USA thats minimize) it? Again, thanks so much! Matt
Re: [reiserfs-list] Silly question, defrag
On Wed, Apr 03, 2002 at 08:08:21AM -0800, Matthew Johnson wrote: On Wednesday 03 April 2002 00:21, Joe Cooper wrote: Don't Well I don't, but when newbies who are used to computing on win32 systems hear that they may not just accept the word don't. Actually its hard to find the reasons exactly why one does not defrag. Depending on the elevator algorithms in Linux (sorry, I'm not very familiar with them), performance can actually be *increased* by some bit of fragmentation. If the data is spread out over the disk, I've heard it allows some elevator algorithms to improve their queuing stategies. I'm pretty sure this is the case on Novell Netware. Unfortunately I don't have enough technical knowledge about elevator algorithms to really know if it helps or hurts us. However, I do know that back in the days when I was just getting into Linux stuff (around 2.0.0-ish era), it was generally said that ext2 driver did basic defrag on write and it limited fragmentation to pathological filesystem usage. Again, not sure that this is still the case. Ross Vandegrift [EMAIL PROTECTED]
Re: [reiserfs-list] Silly question, defrag
On Wed, Apr 03, 2002 at 10:24:37PM +0400, Hans Reiser wrote: There are ways in which Windows is better than Linux. This is one. To think that Linux is better for not having defrag is wishful innacurate thinking. Such is life. Send us $30k and defrag will go into v4.0 instead of 4.1.:) I have to wonder about your motives here, Hans. You are the one who stands to gain by capitalizing on newbie Unix/Linux users misunderstanding of filesystems based on their experience with DOS and Windows and here you are promoting defrag as a feature which puts your FS above others. They have learned to compulsively defrag their disks once a week and you are looking to feed their addiction. I think reiserfs is really great and a defrag/repacker will be nice but the above strikes me as a bit strange. How long has ext2 been around as the stock Linux filesystem? More than long enough for people to have realized whether a defragger would be useful. Yet I can't think of a single distribution (of Linux or Unix in general) that comes with a defragger. Stephen Tweedie wrote one for it but nobody bothers to use it or even to include it with their distro. Please don't perpetuate the idea that good filesystems have/need a defragger. -- Tracy Reed http://www.ultraviolet.org She moves in mysterious ways msg05010/pgp0.pgp Description: PGP signature
Re: [reiserfs-list] Silly question, defrag
snip I have to wonder about your motives here, Hans. You are the one who stands to gain by capitalizing on newbie Unix/Linux users misunderstanding of filesystems based on their experience with DOS and Windows and here you are promoting defrag as a feature which puts your FS above others. They have learned to compulsively defrag their disks once a week and you are looking to feed their addiction. I think reiserfs is really great and a defrag/repacker will be nice but the above strikes me as a bit strange. How long has ext2 been around as the stock Linux filesystem? More than long enough for people to have realized whether a defragger would be useful. Yet I can't think of a single distribution (of Linux or Unix in general) that comes with a defragger. Stephen Tweedie wrote one for it but nobody bothers to use it or even to include it with their distro. Please don't perpetuate the idea that good filesystems have/need a defragger. -- Tracy Reed http://www.ultraviolet.org She moves in mysterious ways I'd actually suggest you are one of the poeple perpetuating a fragmentation myth. Fragmentation is an artifact of random access to disk media. It is caused when the final size of a file is not known by the filesystem layer at write time and hence it is not possible to allocate the total space in one contiguous block. Log files and variable database files that grow over time are a classic example of this scenario. This leaves the the only option to defrag once the file has been completed OR at some user defined interval. Some fs may try to reduce the incidence of this but cant prevent it happening. The response that some fs's that are better than DOS/WinXX and dont have this problem is simply a cop out and serve to confuse the issue. Its these linux is better than winxx statements without a matching technical explanation that discredit us in the eyes of others. Certainly the behavior of DOS/WIN systems in operation (endless upgrades, driver installs etc) promote fragmentation however some linux deployments fall foul to the same problem albeit for a different reason. (We have been bitten by this). Currently the only way to defrag an ext2 fs (or reiserfs) is a copy off, mke2fs and copy back cycle - hardly ideal. BTW if you are interested in scripts that capable of totally framenting an ext2 partition (as reported by chke2fs and demonstrated by timing a grep) and also a discussion of the problem - check the archives. Hans has recognised a genuine (though not popular or talked about) need in some applications of the fs and is fixing it. Good on him. Cheers -Rod
Re: [reiserfs-list] Silly question, defrag
On 04/04/2002 01:49 AM, Tracy R Reed wrote: On Wed, Apr 03, 2002 at 08:08:21AM -0800, Matthew Johnson wrote: On Wednesday 03 April 2002 00:21, Joe Cooper wrote: Don't Well I don't, but when newbies who are used to computing on win32 systems hear that they may not just accept the word don't. Actually its hard to find the reasons exactly why one does not defrag. It is more useful to look at why one DID defrag back in the bad ol' days of DOS and Windows. IIRC, the FAT filesystem would scan through it's equivalient of the free block list and start writing at the first free block. If it wrote for a while and then there was other data in the way it stop and go to the next free space. This way fragmentation was practically guarenteed and it happened rapidly. Modern filesystems use much smarter ways of laying out data on the disk so that fragmentation happens much less often. Now you will almost certainly waste more time by defragmenting than you would suffering whatever performance hit the little fragmentation there is causes. I've been using Linux/Unix for 10 years and I have never (not once!) defragged a filesystem. Perhaps I should aim this message to the kernel mailing list, so that I can get response from a wider array of people who like other filesystems. But its not kernel related. I wouldn't recommend doing that. The answer is pretty much the same regardless of the filesystem. If it's a non-FAT fs you probably don't have to worry about fragmentation. Yesss. I like defragmenters on my Win98 disks, as I really see a speedup after using them (e.g after new software installations OR a longer status quo, but the effect depends on the defragmenters configuration). You describe how/why this makes sense on FAT FS. When I backupped my ReiserFS partitions monthly I used to recreate the original FS if everything was o.k. and copy back the whole content. That's no server here, it's a standalone notebook. After that procedure I found some applications that worked faster and some that were slower than before. O.k. I may have only subjectively compared the load times of NS6 +32MB disk cache and SO5.2. They were different than before copying, but the sum didn't show any advantage. So?: I don't really need a Defragmenter on v3.6 ReiserFS in FAT scales for my usage and I really don't need to recreate and copy-back. Mmh, just wanted to add my experience, best wishes, Manuel
Re: [reiserfs-list] Silly question, defrag
From: [EMAIL PROTECTED] snip I have to wonder about your motives here, Hans. You are the one who stands to gain by capitalizing on newbie Unix/Linux users misunderstanding of filesystems based on their experience with DOS and Windows and here you are promoting defrag as a feature which puts your FS above others. They have learned to compulsively defrag their disks once a week and you are looking to feed their addiction. I think reiserfs is really great and a defrag/repacker will be nice but the above strikes me as a bit strange. How long has ext2 been around as the stock Linux filesystem? More than long enough for people to have realized whether a defragger would be useful. Yet I can't think of a single distribution (of Linux or Unix in general) that comes with a defragger. Stephen Tweedie wrote one for it but nobody bothers to use it or even to include it with their distro. This leaves the the only option to defrag once the file has been completed OR at some user defined interval. Some fs may try to reduce the incidence of this but cant prevent it happening. True. Some of the measures that are taken can be quite effective. Reserving 10% overhead which only root is allowed to write in is extremely effective with SunOS's flavor of UFS. Currently the only way to defrag an ext2 fs (or reiserfs) is a copy off, mke2fs and copy back cycle - hardly ideal. BTW if you are interested in scripts that capable of totally framenting an ext2 partition (as reported by chke2fs and demonstrated by timing a grep) and also a discussion of the problem - check the archives. Not true, there is a e2defrag out there. Now, e2defrag will defragment a filesystem, however the problem is that it greatly promotes future fragmentation (got to maybe 2% fragmentation after months of use, but within a week of running e2defrag it hit 8% fragmentation). I must theorize that it packs files tightly rather than using the headroom leading to the effect seen. -- |\__/|\__/|\__ --= 8-) EHM =-- __/|\__/|\__/| \|| | [EMAIL PROTECTED] PGP 8881EF59 | ||/ \ \ | __| -O #include stddisclaimer.h O- |__ | / / \___\_|/82 04 A1 3C C7 B1 37 2A E3 6E 84 DA 97 4C 40 E6\|_/___/
