Re: Filesystem tuning for lots of small files (a Maildir)?
Craig Reyenga wrote: - Original Message - From: "Darren Pilgrim" To: Sent: Friday, January 24, 2003 07:13 Subject: Filesystem tuning for lots of small files (a Maildir)? I'm currently facing a problem of having used Netscape (now Mozilla) for years in Windows and now trying to find something I can regularly use in FreeBSD without losing Mozilla in Windows. I've mostly settled on IMAP (courier) with procmail filters, but that raises the issue of filesystem performance for directories with large numbers of files/subdirectories in them. I have more than 32,000 emails stored. How do I calculate/see the number of available inodes?^^^ > ^ df -i /filesystem-in-question The filesystem has about 1.4m free inodes, so I guess that's not really going to be a problem. What's the max. number of inodes I can have? 2^32? The existing filesystem was newfs'd with the sysinstall defaults. Should I re-newfs it with different values? What would I want to set them at? I know I'd need to adjust things to make sure I have enough inodes for 40,000+ files, but what about the block and fragment size? Should I use smaller values like 8192/1024 or 4096/512 or is the default 16384/2048 best? Higher values would just increase slack space, right? What are the impacts of lower values? The number of inodes varies with the filesystem size and bytes per inode. So if you're talking about a huge filesystem, you're probably all set as it is. However, I needed a /usr that has many inodes, so I doubled the default by doing this: newfs -b 16384 -f 2048 -i 4096 /usr -i 4096 is half as many bytes per inode compared to the default 8192, therefore, I have 2X as many inodes. See newfs(8) for more info. tuning(7) also. I know that one inode is used for every file (for arguement's sake we'll say everything that uses an inode is a file) in a filesystem. So the number of inodes is the number of files you can have. But what happens when the file is larger than the inode size? It still uses one inode, but the filesystem has to allocate space in blocks. I'm trying to determine the size of that block, if it's adjustable, and if I even should be adjusting it. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-questions" in the body of the message
Re: Filesystem tuning for lots of small files (a Maildir)?
- Original Message - From: "Darren Pilgrim" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, January 24, 2003 07:13 Subject: Filesystem tuning for lots of small files (a Maildir)? > I'm currently facing a problem of having used Netscape (now Mozilla) for > years in Windows and now trying to find something I can regularly use in > FreeBSD without losing Mozilla in Windows. > > I've mostly settled on IMAP (courier) with procmail filters, but that > raises the issue of filesystem performance for directories with large > numbers of files/subdirectories in them. I have more than 32,000 emails > stored. How do I calculate/see the number of available inodes? ^ df -i /filesystem-in-question > The > existing filesystem was newfs'd with the sysinstall defaults. Should I > re-newfs it with different values? What would I want to set them at? I > know I'd need to adjust things to make sure I have enough inodes for > 40,000+ files, but what about the block and fragment size? Should I use > smaller values like 8192/1024 or 4096/512 or is the default 16384/2048 > best? Higher values would just increase slack space, right? What are > the impacts of lower values? > The number of inodes varies with the filesystem size and bytes per inode. So if you're talking about a huge filesystem, you're probably all set as it is. However, I needed a /usr that has many inodes, so I doubled the default by doing this: newfs -b 16384 -f 2048 -i 4096 /usr -i 4096 is half as many bytes per inode compared to the default 8192, therefore, I have 2X as many inodes. See newfs(8) for more info. tuning(7) also. > Some folders, like the one for the postfix-users list, can have > 3000-4000 messages in them. For growth, we'll say 5000 messages. The > IMAP layout with Courier means all the folders sit all on one level > under ~/Maildir, which means I'd have 200 or so subdirectories in one > place. I have the UFS_DIRHASH option enabled for the my MP3 collection, > but that's as case of 300 subdirecories in one directory, not 5000 > files. What else can I do to tune for this kind of (ab)use? > Not sure. I hope at least part of this message was somewhat-kinda-sorta-maybe helpful. [Snipping mail questions; I have no idea.] -Craig To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-questions" in the body of the message
Filesystem tuning for lots of small files (a Maildir)?
I'm currently facing a problem of having used Netscape (now Mozilla) for years in Windows and now trying to find something I can regularly use in FreeBSD without losing Mozilla in Windows. I've mostly settled on IMAP (courier) with procmail filters, but that raises the issue of filesystem performance for directories with large numbers of files/subdirectories in them. I have more than 32,000 emails stored. How do I calculate/see the number of available inodes? The existing filesystem was newfs'd with the sysinstall defaults. Should I re-newfs it with different values? What would I want to set them at? I know I'd need to adjust things to make sure I have enough inodes for 40,000+ files, but what about the block and fragment size? Should I use smaller values like 8192/1024 or 4096/512 or is the default 16384/2048 best? Higher values would just increase slack space, right? What are the impacts of lower values? Some folders, like the one for the postfix-users list, can have 3000-4000 messages in them. For growth, we'll say 5000 messages. The IMAP layout with Courier means all the folders sit all on one level under ~/Maildir, which means I'd have 200 or so subdirectories in one place. I have the UFS_DIRHASH option enabled for the my MP3 collection, but that's as case of 300 subdirecories in one directory, not 5000 files. What else can I do to tune for this kind of (ab)use? P.S. I really would like to stick with Maildirs and Courier-IMAP for this. I know CIMAP well and it has proven very fast and stable for what I do with it. However, if these demands are just too much to expect from an IMAP-accessed Maildir, Courier, or FreeBSD, what are my alternatives? To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-questions" in the body of the message
