Re: Maildir performance & mutt (was Re: Someone tell me the secret of mutt)
on Fri, Apr 12, 2002, Rich Puhek ([EMAIL PROTECTED]) wrote: > > dman wrote: > > > Karsten wrote: > > | With 1000+ messages (and 5000+ is pretty easily attainable), performance > > | on opening a folder sucks. I assume it's because mutt has to do an > > | fopen() on each file, scan headers, group output, and sort it. The > > | result is a several-seconds (sometimes 10-20) on opening large folders. > > More memory for disk cache? I've found that the first time I open a <...> > I'm guessing it's the sheer number of messages in the directory... ext2 I'm using Reiserfs. The problem doesn't seem to be scanning the filelist, but opening each file to read the mail headers. Peace. -- Karsten M. Selfhttp://kmself.home.netcom.com/ What Part of "Gestalt" don't you understand? What doesn't kill you makes you stranger. pgphsRT4ln3K2.pgp Description: PGP signature
Re: Maildir performance & mutt
* Karsten M. Self [2002-04-12 00:25:00 -0700]: > > I've switched to maildir, um, well, I forget why. > > With 1000+ messages (and 5000+ is pretty easily attainable), performance > on opening a folder sucks. > > Is there any way to speed this up? There was a thread about this recently on mutt-users. Here is good explanation of the available solutions from Thomas Hurst: > > Maildir is slower at being opened; the simple act of opening a file, > > scanning the first few bytes, then closing it 3500 times is always going > > to be slower than simply opening one file and seeking through it, unless > > your filesystem is really incredibly good at organising it to minimise > > seeking and give a miniscule overhead to the extra syscalls. That's > > without mentioning having to take directory listings of two directories > > beforehand. > > > > The solutions are: > > > > 1. Switch to mbox and trade off individual mail modification speed and > > corruption resistance for initial opening speed. > > > > 2. Use a maildir caching patch to limit scanning of new messages to > > operations on a dbm. > > > > 3. Make use of the low cost of moving messages from the start of the > > maildir to archive old messages. Leave your working folder as maildir > > with a maximum of a couple of days mails and keep mbox archives or so. > > > > 4. Find a filesystem which keeps lots of small files in the same dir > > consolidated together with the metainformation it needs to find them to > > cut down seeks and small reads. > > > > 5. Get tonnes of memory and try to keep as much as possible of it > > cached. On FreeBSD this tends to cut opening time to about 10% slower > > than mbox. Some people use solution #2, Michael Elkins' header caching patch for maildir: http://www.cs.hmc.edu/~me/mutt/ and they have been very satisfied with the results. I believe you need to patch it against the cvs development branch of Mutt, so if you are against that sort of thing, just wait till it gets merged in and released, but I don't know how long that will take. -- N. Thomas [EMAIL PROTECTED] Etiamsi occiderit me, in ipso sperabo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Maildir performance & mutt (was Re: Someone tell me the secret of mutt)
dman wrote: > > | With 1000+ messages (and 5000+ is pretty easily attainable), performance > | on opening a folder sucks. I assume it's because mutt has to do an > | fopen() on each file, scan headers, group output, and sort it. The > | result is a several-seconds (sometimes 10-20) on opening large folders. > | > | Is there any way to speed this up? > > More memory for disk cache? I've found that the first time I open a > large maildir folder the disk will crank for a while. I expect it > would do just as bad if it was mbox, probably worse. The next time it > is real fast because the disk is cached in memory. At any rate, > updating the folder (ie deleting one message) is much faster. > I'm guessing it's the sheer number of messages in the directory... ext2 starts to get kinda slow when you go over 1000 messages. This was the first reason I switched to Maildirs BTW... had a server with about 3000 users. Access to /var/spool/mail was very slow as a result... I'm hoping ReiserFS starts to solidify as a file system... That's not supposed to have the performance hit with 1000s of files in one directory, but I still don't trust it for a mail spool. I'm eventually going to try reiser for the host status directory... that's another area that gets a large number of files, and I don't care if it gets corrupted (worst case I'll just wipe it out and let it re-populate). --Rich _ Rich Puhek ETN Systems Inc. _ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Maildir performance & mutt (was Re: Someone tell me the secret of mutt)
On Fri, Apr 12, 2002 at 12:25:00AM -0700, Karsten M. Self wrote: | on Thu, Apr 11, 2002, dman ([EMAIL PROTECTED]) wrote: | | > You have no lack of control over how it behaves. (I use maildir | > myself) | | OK, question for you. | | I've switched to maildir, um, well, I forget why. Maybe peformance, | maybe trepidation over my mbox files breaking one day. Yeah, the lack of a lock is great. | With 1000+ messages (and 5000+ is pretty easily attainable), performance | on opening a folder sucks. I assume it's because mutt has to do an | fopen() on each file, scan headers, group output, and sort it. The | result is a several-seconds (sometimes 10-20) on opening large folders. | | Is there any way to speed this up? More memory for disk cache? I've found that the first time I open a large maildir folder the disk will crank for a while. I expect it would do just as bad if it was mbox, probably worse. The next time it is real fast because the disk is cached in memory. At any rate, updating the folder (ie deleting one message) is much faster. | The faster of the two boxes is a PIII-400 (faster's on the way), 384 | MiB, SCSI drive. Duron 750 , 2 or 3 year old IDE disk , 256 MB RAM -D -- Many a man claims to have unfailing love, but a faithful man who can find? Proverbs 20:6 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Maildir performance & mutt (was Re: Someone tell me the secret of mutt)
On Fri, 12 Apr 2002 10:02:46 -0700 Craig Dickson <[EMAIL PROTECTED]> wrote: > Sylpheed opens folders very quickly because it keeps its own index > file in each folder, and so does not need to read headers from every > file. Yes. :) > Since Sylpheed uses MH format, which requires a folder lock any > time you want to modify a folder, it can do this without any real > disadvantage. Maildir, however, does not need folder locks, so adding > an index file(which would have to be locked for modification) seems > like a bad idea. Well, I didn't think that MH required entire-directory locks. Anyways, even if that's true, a) Mutt doesn't use a cache, even for MH, b) Mutt doesn't use a cache, for anything(I don't think), and c) there's no reason the cache needs to be within the folder itself :) -- \ David B. Harris, Systems administrator | http://www.terrabox.com / / [EMAIL PROTECTED], [EMAIL PROTECTED] | http://eelf.ddts.net \ \==/ / Clan Barclay motto: Aut agere, aut mori. (Either action, or death.) \ pgpgubye1MdnX.pgp Description: PGP signature
Re: Maildir performance & mutt (was Re: Someone tell me the secret of mutt)
begin David B Harris quotation: > P.S.: Fastest MUA I can bear? Sylpheed. It bites that it's GTK+ and not > console/text, but despite that it's still the fastest fullscreen/gui > mailer I found, and is faster than many CLI(mail, nmh, etc) clients to > boot. Sylpheed opens folders very quickly because it keeps its own index file in each folder, and so does not need to read headers from every file. Since Sylpheed uses MH format, which requires a folder lock any time you want to modify a folder, it can do this without any real disadvantage. Maildir, however, does not need folder locks, so adding an index file (which would have to be locked for modification) seems like a bad idea. Craig pgpmD0sNvThiZ.pgp Description: PGP signature
Re: Maildir performance & mutt (was Re: Someone tell me the secret of mutt)
begin Pedro Zorzenon Neto quotation: > I had problems in mailboxes with more then 5000 messages. They > were very slow to open. > > I solved it with a procmail rule that timestamps it... > > Now I have monthly taged mailboxes. (Spool-2002-01, Spool-2002-02...) > > I know it is not what you asked, but it can solve your problem. I do this (on an annual rather than monthly basis) for a few mailing lists. The few lists I'm on that generate enough traffic that monthly archives would be really big, I simply don't archive locally. Most of them have web archives anyway (debian-user, etc.). My debian-user folder has maybe 50 messages in it that I've found valuable enough to want to keep around semi-permanently. The rest get deleted manually after I read them, or after their threads have died. Craig pgpMNX8vZJQy1.pgp Description: PGP signature
Re: Maildir performance & mutt (was Re: Someone tell me the secret of mutt)
On Fri, 12 Apr 2002 00:25:00 -0700 "Karsten M. Self" wrote: > I've switched to maildir, um, well, I forget why. Maybe peformance, > maybe trepidation over my mbox files breaking one day. > > With 1000+ messages (and 5000+ is pretty easily attainable), > performance on opening a folder sucks. I assume it's because mutt has > to do an fopen() on each file, scan headers, group output, and sort > it. The result is a several-seconds (sometimes 10-20) on opening > large folders. > > Is there any way to speed this up? > > The faster of the two boxes is a PIII-400 (faster's on the way), 384 > MiB, SCSI drive. Hey there :) I typically read huge volumes of email (I tend to read a lot of lists) ... maybe five, six hundred emails a day. Anyways, you can imagine that over time my mailstore gets to be quite big ;) At the same time, performance is a serious issue for me. If it takes more than two or three seconds to get into any single mailbox, I'm grinding my teeth. So, with that, I went around poking at all the various MUAs around, doing some simple benchmarks with a stopwatch. Mutt was pretty slow. I mean, more than fast enough for most people, but it isn't going to win any awards for speed. I tested with MH maildir, Qmail Maildir, mbox and imap. It was all fairly slow. Unpleasantly so, unfortunatly. Anyways, I spent a lot of time trying to make it faster (because, really, I do like Mutt), but it just didn't happen. Spent about two months asking around too, with no luck. As Pedro said, the only option is to reduce the size of your mailboxes. :) P.S.: Fastest MUA I can bear? Sylpheed. It bites that it's GTK+ and not console/text, but despite that it's still the fastest fullscreen/gui mailer I found, and is faster than many CLI(mail, nmh, etc) clients to boot. -- \ David B. Harris, Systems administrator | http://www.terrabox.com / / [EMAIL PROTECTED], [EMAIL PROTECTED] | http://eelf.ddts.net \ \==/ / Clan Barclay motto: Aut agere, aut mori. (Either action, or death.) \ pgp2MIADiObCm.pgp Description: PGP signature
Re: Maildir performance & mutt (was Re: Someone tell me the secret of mutt)
On Fri, Apr 12, 2002 at 12:25:00AM -0700, Karsten M. Self wrote: > Is there any way to speed this up? Hi Karsten, I had problems in mailboxes with more then 5000 messages. They were very slow to open. I solved it with a procmail rule that timestamps it... Now I have monthly taged mailboxes. (Spool-2002-01, Spool-2002-02...) I know it is not what you asked, but it can solve your problem. Regards, Pedro Sample procmail file: --- begin --- PATH=$HOME/bin:/usr/bin:/bin:/usr/local/bin:. MAILDIR=$HOME/Mail # You'd better make sure it exists DEFAULT=$MAILDIR/Spool-`date +%Y-%m` LOGFILE=$MAILDIR/from-`date +%Y-%m` LOCKFILE=$HOME/.lockmail :0: * ^X-Mailing-List:.*debian-user\@ deb-user-`date +%Y-%m` :0: * ^X-Mailing-List:.*some-mailing-list\@ low-trafic-mailing-list-`date +%Y` :0: * ^X-Mailing-List:.*other-mailing-list\@ very-high-trafic-mailing-list-`date +%Y-%U` --- end --- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Maildir performance & mutt (was Re: Someone tell me the secret of mutt)
on Thu, Apr 11, 2002, dman ([EMAIL PROTECTED]) wrote: > You have no lack of control over how it behaves. (I use maildir > myself) OK, question for you. I've switched to maildir, um, well, I forget why. Maybe peformance, maybe trepidation over my mbox files breaking one day. With 1000+ messages (and 5000+ is pretty easily attainable), performance on opening a folder sucks. I assume it's because mutt has to do an fopen() on each file, scan headers, group output, and sort it. The result is a several-seconds (sometimes 10-20) on opening large folders. Is there any way to speed this up? The faster of the two boxes is a PIII-400 (faster's on the way), 384 MiB, SCSI drive. Peace. -- Karsten M. Selfhttp://kmself.home.netcom.com/ What Part of "Gestalt" don't you understand? There is no K5 Cabal: http://www.kuro5hin.org/ pgpcI67mgSByq.pgp Description: PGP signature
