On Thu, 2007-11-01 at 16:11 -0400, dave crane wrote: > I'd completely forgotten about an issue we had a ways back that might > be related. > > About two years ago we internally identified a JFS oddity we coined > "STP directories" (because they were dead and bloated). We use a > number of temporary directories for a cluster where 5-250 100b-50k > files per second are created, and shortly after read and destroyed. > The number of files in the directory at any given time is 100-5000. > > After a while, not only did ls and find take a long time, but > eventually the directory just stopped. Any attempt to stat, touch, > find, read, write, delete, etc would just hang. > > An ls in the folder directly above showed the directory size as > somewhere around 70mb. Not the contents, folder itself. We didn't > have the time to dig through the source, so we just moved the > directory out of the way and made a new one. A copy eventually got > all the files out of the "big" folders. > > We theorized at the time (again, without looking at the source, so > this is coming straight out of my @ss), that the as files were being > deleted the directory entries were not getting deleted/reused > completely so blocks kept getting allocated, and never got > deallocated. fsck gave nothing back.
Part of the directory is a table that slowly grows, and deleted entries are not reused. (The deleted entries actually still contain a pointer to the next entry at the time the file was deleted. This allows directory traversals to resume at the correct place while entries are being deleted. This fixed the infamous "rm -rf" problem in the early jfs days.) There is a bit of overhead maintaining this table as files are added and deleted from the directory, but I wouldn't expect lookups to be affected by it (ls or stat). The biggest downfall is wasted space. I really should come up with some mechanism to reuse the old slots after some time, but it will be non-trivial to fix. Another option would be to have fsck fix the problem if the number of unused entries is excessive. > > The directory swap is now scripted to happen once a week. We actually > re-wrote a bit of our app to use either directory so we could move the > directory and create a new one and not delete the old one until all of > the files had been processed. The really high volume directory > (100-250 files per second) were broken up into per-minute > subdirectories. This solution was acceptable to us and I forgot to > report it. If the directory becomes completely empty, it shouldn't be necessary to delete and re-create it. Removing the last entry from a directory frees the table. > > We had moved all our folders like this off ext2 around the time JFS > made it into mainline. We've experimented with migrating off JFS to > ext3 or XFS, but other issues always crop up. In fact, head to head > our 2 year old JFS filesystems still beat _new_ XFS, ext3 and reiser4 > in our application. > > Other than this the bloat, which I admit is a result of an extreme > use, JFS just works. It uses less CPU, is stable, has never been > damaged past the point of fsck repair, and easily resizes. I don't > think we've ever lost a file. We just have some extras we can't > delete in these old STP directories :) I'm not quite sure what to make of your STP directories. I don't know what would cause a hang that fsck wouldn't catch. > > The only this other than this I'd like to see changed is the default > journal size calculation. 0.4% of 1TB+ is a bit big, and makes normal > log recoveries/checks feel like the bad old ext2 days. A static > journal size over a certain filesystem size would seem to make sense, > and balance what I assume was the original intent of the percentage. The size is capped at 32 MB, or at least it should be. (I guess the man page doesn't tell you this.) Are you seeing a larger journal, or is 32 MB still too big? > > Anything I can do to help with this, please let me know. You still have some STP directories? Maybe we can learn something about them, and fix whatever the problem is. -- David Kleikamp IBM Linux Technology Center ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Jfs-discussion mailing list Jfs-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jfs-discussion
