Re: header_cache for mbox
On 16.07.2015, Heinz Diehl wrote: I'll recompile without hcache and report back. Ok, done. Still, hcache is a huge improvement in the way I reported in my other mails.
Re: header_cache for mbox
On Thu, Jul 16, 2015 at 05:37:46PM -0700, Ian Zimmerman wrote: On 2015-07-16 18:50 -0500, Derek Martin wrote: for lseek() to be useful, you need to know where to lseek to which you wouldn't in this case (if you want reliable parsing). That's easy though, obviously you need to fully parse unread messages, but once they're parsed, you just cache the offsets. What about deleting messages? What about editing (which mutt allows) or rethreading? Any of these actions will invalidate the offsets. Delete may or may not (it just sets a flag--I've long thought that mbox should have a fixed-width flag header so this can just be overwritten in place); but that need not necessarily be written out until the user syncs--a natural time to do a reparse. Expunging messages clearly would, as well as editing... but those operations are relatively rare. You'd need to reparse anyway, so... just do that. But you only need to reparse from the first modified message, so keeping the cache saves you some work even in that case. With mbox's typical use cases, it's far more likely that you'll modify messages near the end of the file than near the beginning, so usually this won't be a very costly operation. With very large mbox mailboxes, the time savings would be HUGE for typical usage patterns. The downside is, it's a lot more complex. -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. pgp8rPb0s84TZ.pgp Description: PGP signature
Re: header_cache for mbox
On 15.07.2015, Will Yardley wrote: Did you time the load with header caching disabled as well? No, in fact I didn't. I've found that opening an uncached folder, or a folder where the cache is very out of date, sometimes can take a bit longer (while the cache is being built, presumably), but loading it again tends to be faster. I'll recompile without hcache and report back.
Re: header_cache for mbox
On Thu, Jul 16, 2015 at 08:00:34AM +0200, Heinz Diehl wrote: On 15.07.2015, Will Yardley wrote: Did you time the load with header caching disabled as well? No, in fact I didn't. I've found that opening an uncached folder, or a folder where the cache is very out of date, sometimes can take a bit longer (while the cache is being built, presumably), but loading it again tends to be faster. I'll recompile without hcache and report back. I think you don't need to compile without it; just unset $header_cache. Of course, db library linked against and filesystem will affect the speed of reading the mailbox and building the cache, so the results may differ drastically from person to person. w
Re: header_cache for mbox
On 2015-07-16 18:50 -0500, Derek Martin wrote: for lseek() to be useful, you need to know where to lseek to which you wouldn't in this case (if you want reliable parsing). That's easy though, obviously you need to fully parse unread messages, but once they're parsed, you just cache the offsets. What about deleting messages? What about editing (which mutt allows) or rethreading? Any of these actions will invalidate the offsets. -- Please *no* private copies of mailing list or newsgroup messages. Rule 420: All persons more than eight miles high to leave the court.
Re: header_cache for mbox
On Wed, Jul 15, 2015 at 09:34:25AM +1000, m...@raf.org wrote: Derek Martin wrote: On Tue, Jul 14, 2015 at 08:23:25PM +0200, Heinz Diehl wrote: On 14.07.2015, Kevin J. McCarthy wrote: With mbox, I guess the designers thought there wouldn't be that much of a speed improvement because it's just a sequential read of a single file. That sounds reasonable. Except, as far as I can tell, it isn't. I see no reason hcache could not significantly speed up scanning mbox folders as well, at least on any system that supports lseek() or similar (which I imagine is any system that Mutt runs on currently). The amount of benefit you'd get from this would greatly depend on the nature of the messages stored in the folder, though... Folders of moderate size or larger, with mostly large messages (or attachments) should see the most benefit, and those with many small messages, or with very few messages, would see the least (but still some). for lseek() to be useful, you need to know where to lseek to which you wouldn't in this case (if you want reliable parsing). That's easy though, obviously you need to fully parse unread messages, but once they're parsed, you just cache the offsets. You might need to reparse of the thing at the offset you've cached doesn't look like a message (doesn't start with From ) but otherwise you should be good. and anyway, i'd like to think mutt uses mmap() for mbox files. I'd like to think that too, but it doesn't. ;-) Or at least, my recollection is that it doesn't. I brought this up in the (distant) past, and IIRC the response was that too many systems had a broken mmap and that it could be underperforming file I/O for other reasons. Which sounded like a crock to me, but I let it go at that, since I didn't want to rewrite the mbox code. -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. pgpdkyyMe15L3.pgp Description: PGP signature
Re: header_cache for mbox
On 15.07.2015, Ian Zimmerman wrote: I'm using both IMAP and maildir, and hcache is a real and quite noticeable improvement. FWIW, I use maildir too but locally, and turning hcache on did not result in a clear speed up. My largest folders are around 5000 messages each. I have mailboxes from ~1000 up to 20.000 emails and the hcache speedup is clearly noticeable with all of them. I just deleted the cache file and compared the speed of the first mailbox read to the cached one. It's a huge improvement. Btw: which hcache backend do you use? I'm using GDBM, and hcache is really extremly fast.
Re: header_cache for mbox
On Wed, Jul 15, 2015 at 10:51:32PM +0200, Heinz Diehl wrote: On 15.07.2015, Ian Zimmerman wrote: I'm using both IMAP and maildir, and hcache is a real and quite noticeable improvement. FWIW, I use maildir too but locally, and turning hcache on did not result in a clear speed up. My largest folders are around 5000 messages each. I have mailboxes from ~1000 up to 20.000 emails and the hcache speedup is clearly noticeable with all of them. I just deleted the cache file and compared the speed of the first mailbox read to the cached one. It's a huge improvement. Did you time the load with header caching disabled as well? I've found that opening an uncached folder, or a folder where the cache is very out of date, sometimes can take a bit longer (while the cache is being built, presumably), but loading it again tends to be faster. w
Re: header_cache for mbox
I have the same compile flags, however I opted for the hcache backend as tokyocabinet. Even with GDBM, I could not implement header caching on mbox :( In fact I thought at one time it was working with mbox as the loading was apparently faster than without the header_cache set in config. But I could not reproduce it. So is this indeed supported for mbox? It works for me consistantly with imap and with maildir. Faster and the files in the header_cache dir are uopdated. Nope on both counts with mbox. What is the reason for this? Eric ~ % mutt -v Mutt 1.5.23 (2014-03-12) Copyright (C) 1996-2009 Michael R. Elkins and others. Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'. Mutt is free software, and you are welcome to redistribute it under certain conditions; type `mutt -vv' for details. System: Linux 3.19.0-22-generic (i686) ncurses: ncurses 5.9.20140712 (compiled with 5.9) libidn: 1.28 (compiled with 1.28) hcache backend: tokyocabinet 1.4.48 Compiler: Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/i686-linux-gnu/4.9/lto-wrapper Target: i686-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.9.2-5ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-i386/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-i386 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-i386 --with-arch-directory=i386 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-targets=all --enable-multiarch --disable-werror --with-arch-32=i686 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=i686-linux-gnu --host=i686-linux-gnu --target=i686-linux-gnu Thread model: posix gcc version 4.9.2 (Ubuntu 4.9.2-5ubuntu1) Configure options: '--prefix=/usr' '--sysconfdir=/etc' '--mandir=/usr/share/man' '--with-docdir=/usr/share/doc' '--with-mailpath=/var/mail' '--disable-dependency-tracking' '--enable-compressed' '--enable-debug' '--enable-fcntl' '--enable-hcache' '--enable-gpgme' '--enable-imap' '--enable-smtp' '--enable-pop' '--with-curses' '--with-gnutls' '--with-gss' '--with-idn' '--with-mixmaster' '--with-sasl' '--without-gdbm' '--without-bdb' '--without-qdbm' '--build' 'i686-linux-gnu' '--enable-nntp' 'build_alias=i686-linux-gnu' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'CPPFLAGS=-D_FORTIFY_SOURCE=2 -I/usr/include/qdbm' Compilation CFLAGS: -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall Compile options: -DOMAIN +DEBUG -HOMESPOOL +USE_SETGID +USE_DOTLOCK +DL_STANDALONE +USE_FCNTL -USE_FLOCK +USE_POP +USE_NNTP +USE_IMAP +USE_SMTP -USE_SSL_OPENSSL +USE_SSL_GNUTLS +USE_SASL +USE_GSS +HAVE_GETADDRINFO +HAVE_REGCOMP -USE_GNU_REGEX +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM +CRYPT_BACKEND_CLASSIC_PGP +CRYPT_BACKEND_CLASSIC_SMIME +CRYPT_BACKEND_GPGME -EXACT_ADDRESS -SUN_ATTACHMENT +ENABLE_NLS -LOCALES_HACK +COMPRESSED +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR +HAVE_ICONV -ICONV_NONTRANS +HAVE_LIBIDN +HAVE_GETSID +USE_HCACHE -ISPELL SENDMAIL=/usr/sbin/sendmail MAILPATH=/var/mail PKGDATADIR=/usr/share/mutt SYSCONFDIR=/etc EXECSHELL=/bin/sh MIXMASTER=mixmaster To contact the developers, please mail to mutt-...@mutt.org. To report a bug, please visit http://bugs.mutt.org/. misc/am-maintainer-mode.patch features/ifdef.patch snip Patrick Shanahan wrote on Mon-13-Jul 15 9:10PM * Heinz Diehl htd...@fritha.org [07-13-15 14:42]: On 13.07.2015, Patrick Shanahan wrote: * Eric Smith e...@fruitcom.com [07-13-15 09:17]: Has anyone implemented this or is considering to do so? Been using it for *quite* a few years now. AFAIK, mutt's header cache only works for IMAP and Maildir, but not for the mbox format. Guess more research is warranted. I have used mbox for 10 years and header_cache since implementation on openSUSE. user:~ mutt -v Mutt 1.5.23 (2014-03-12) Copyright (C) 1996-2009 Michael R. Elkins and others. Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'. Mutt is free software, and you are welcome to redistribute it under certain conditions; type `mutt -vv' for details. System: Linux
Re: header_cache for mbox
Eric Smith wrote: So is this indeed supported for mbox? Sorry, hcache is only implemented for pop, imap, maildir, and mh. What is the reason for this? I think the hcache helps to alleviate network latency (pop/imap) and for opening and closing thousands of files (maildir/mh). With mbox, I guess the designers thought there wouldn't be that much of a speed improvement because it's just a sequential read of a single file. -- Kevin J. McCarthy GPG Fingerprint: 8975 A9B3 3AA3 7910 385C 5308 ADEF 7684 8031 6BDA http://www.8t8.us/configs/gpg-key-transition-statement.txt signature.asc Description: PGP signature
Re: header_cache for mbox
[14/07 12:59] Jon LaBadie: On Tue, Jul 14, 2015 at 05:29:23PM +0100, Larry Hynes wrote: [14/07 12:42] Eric Smith: It works for me consistantly with imap and with maildir. Faster and the files in the header_cache dir are uopdated. Nope on both counts with mbox. What is the reason for this? From the manual: Mutt provides optional support for caching message headers for the following types of folders: IMAP, POP, Maildir and MH. mbox is not mentioned, presumably because caching of message headers is not supported for the mbox format? I'm confused. mbox is a storage format. POP is a message transfer protocol. I use POP to transfer from an mbox file on the server to an mbox file on my desktop. Is caching supported because I use POP or not supported because I use mbox? Well, there lies a quandary for the ages. The wiki doesn't mention POP as a folder format, so presumably it is an impostor at the feet of which you may lay your confusion. The wiki does, however, provide one more small clue for the OP... If you happend (sic) to choose either MH or Maildir, you can improve performance by using header caching... http://dev.mutt.org/trac/wiki/FolderFormat
Re: header_cache for mbox
[14/07 12:42] Eric Smith: I have the same compile flags, however I opted for the hcache backend as tokyocabinet. Even with GDBM, I could not implement header caching on mbox :( In fact I thought at one time it was working with mbox as the loading was apparently faster than without the header_cache set in config. But I could not reproduce it. So is this indeed supported for mbox? It works for me consistantly with imap and with maildir. Faster and the files in the header_cache dir are uopdated. Nope on both counts with mbox. What is the reason for this? From the manual: Mutt provides optional support for caching message headers for the following types of folders: IMAP, POP, Maildir and MH. mbox is not mentioned, presumably because caching of message headers is not supported for the mbox format? Eric ~ % mutt -v Mutt 1.5.23 (2014-03-12) Copyright (C) 1996-2009 Michael R. Elkins and others. Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'. Mutt is free software, and you are welcome to redistribute it under certain conditions; type `mutt -vv' for details. System: Linux 3.19.0-22-generic (i686) ncurses: ncurses 5.9.20140712 (compiled with 5.9) libidn: 1.28 (compiled with 1.28) hcache backend: tokyocabinet 1.4.48 Compiler: Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/i686-linux-gnu/4.9/lto-wrapper Target: i686-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.9.2-5ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-i386/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-i386 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-i386 --with-arch-directory=i386 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-targets=all --enable-multiarch --disable-w error --with-arch-32=i686 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=i686-linux-gnu --host=i686-linux-gnu --target=i686-linux-gnu Thread model: posix gcc version 4.9.2 (Ubuntu 4.9.2-5ubuntu1) Configure options: '--prefix=/usr' '--sysconfdir=/etc' '--mandir=/usr/share/man' '--with-docdir=/usr/share/doc' '--with-mailpath=/var/mail' '--disable-dependency-tracking' '--enable-compressed' '--enable-debug' '--enable-fcntl' '--enable-hcache' '--enable-gpgme' '--enable-imap' '--enable-smtp' '--enable-pop' '--with-curses' '--with-gnutls' '--with-gss' '--with-idn' '--with-mixmaster' '--with-sasl' '--without-gdbm' '--without-bdb' '--without-qdbm' '--build' 'i686-linux-gnu' '--enable-nntp' 'build_alias=i686-linux-gnu' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'CPPFLAGS=-D_FORTIFY_SOURCE=2 -I/usr/include/qdbm' Compilation CFLAGS: -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall Compile options: -DOMAIN +DEBUG -HOMESPOOL +USE_SETGID +USE_DOTLOCK +DL_STANDALONE +USE_FCNTL -USE_FLOCK +USE_POP +USE_NNTP +USE_IMAP +USE_SMTP -USE_SSL_OPENSSL +USE_SSL_GNUTLS +USE_SASL +USE_GSS +HAVE_GETADDRINFO +HAVE_REGCOMP -USE_GNU_REGEX +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM +CRYPT_BACKEND_CLASSIC_PGP +CRYPT_BACKEND_CLASSIC_SMIME +CRYPT_BACKEND_GPGME -EXACT_ADDRESS -SUN_ATTACHMENT +ENABLE_NLS -LOCALES_HACK +COMPRESSED +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR +HAVE_ICONV -ICONV_NONTRANS +HAVE_LIBIDN +HAVE_GETSID +USE_HCACHE -ISPELL SENDMAIL=/usr/sbin/sendmail MAILPATH=/var/mail PKGDATADIR=/usr/share/mutt SYSCONFDIR=/etc EXECSHELL=/bin/sh MIXMASTER=mixmaster To contact the developers, please mail to mutt-...@mutt.org. To report a bug, please visit http://bugs.mutt.org/. misc/am-maintainer-mode.patch features/ifdef.patch snip Patrick Shanahan wrote on Mon-13-Jul 15 9:10PM * Heinz Diehl htd...@fritha.org [07-13-15 14:42]: On 13.07.2015, Patrick Shanahan wrote: * Eric Smith e...@fruitcom.com [07-13-15 09:17]: Has anyone implemented this or is considering to do so? Been using it for *quite* a few years now. AFAIK, mutt's header cache only works for IMAP and Maildir, but not for the mbox format. Guess more research is warranted. I have used mbox for 10 years and header_cache since implementation on openSUSE. user:~ mutt -v Mutt 1.5.23 (2014-03-12) Copyright
Re: header_cache for mbox
On Tue, Jul 14, 2015 at 05:29:23PM +0100, Larry Hynes wrote: [14/07 12:42] Eric Smith: I have the same compile flags, however I opted for the hcache backend as tokyocabinet. Even with GDBM, I could not implement header caching on mbox :( In fact I thought at one time it was working with mbox as the loading was apparently faster than without the header_cache set in config. But I could not reproduce it. So is this indeed supported for mbox? It works for me consistantly with imap and with maildir. Faster and the files in the header_cache dir are uopdated. Nope on both counts with mbox. What is the reason for this? From the manual: Mutt provides optional support for caching message headers for the following types of folders: IMAP, POP, Maildir and MH. mbox is not mentioned, presumably because caching of message headers is not supported for the mbox format? I'm confused. mbox is a storage format. POP is a message transfer protocol. I use POP to transfer from an mbox file on the server to an mbox file on my desktop. Is caching supported because I use POP or not supported because I use mbox? jl -- Jon H. LaBadie j...@jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (703) 935-6720 (C)
Re: header_cache for mbox
On 14.07.2015, Kevin J. McCarthy wrote: I think the hcache helps to alleviate network latency (pop/imap) and for opening and closing thousands of files (maildir/mh). Jepp! And for IMAP, it may not be neccessary at all to read more than the header of most of the mails. Ex: a large mailing list. All headers get cached, making it a breeze to scroll through thousands of mails. Then, only those mails really being read will be fetched. I'm using both IMAP and maildir, and hcache is a real and quite noticeable improvement. With mbox, I guess the designers thought there wouldn't be that much of a speed improvement because it's just a sequential read of a single file. That sounds reasonable.
Re: header_cache for mbox
On Tuesday 14 Jul 2015 20:36:05 Ian Zimmerman wrote: On 2015-07-14 20:23 +0200, Heinz Diehl wrote: I'm using both IMAP and maildir, and hcache is a real and quite noticeable improvement. FWIW, I use maildir too but locally, and turning hcache on did not result in a clear speed up. My largest folders are around 5000 messages each. Hmm ... I have tried to set header_cache on by specifying the path: set header_cache = /var/tmp/.mutt/gmail/header_cache BUT, It takes up to 55 seconds or more to sync the cache everytime I login on my gmail account, downloading 5.4MB of data each time. I have in excess of 125k messages and do not want to download all of these locally. Other smaller IMAP4 accounts are much faster. Is this how it should be, or am I doing something wrong? -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: header_cache for mbox
On 2015-07-14 20:23 +0200, Heinz Diehl wrote: I'm using both IMAP and maildir, and hcache is a real and quite noticeable improvement. FWIW, I use maildir too but locally, and turning hcache on did not result in a clear speed up. My largest folders are around 5000 messages each. -- Please *no* private copies of mailing list or newsgroup messages. Rule 420: All persons more than eight miles high to leave the court.
Re: header_cache for mbox
On Tue, Jul 14, 2015 at 08:23:25PM +0200, Heinz Diehl wrote: On 14.07.2015, Kevin J. McCarthy wrote: With mbox, I guess the designers thought there wouldn't be that much of a speed improvement because it's just a sequential read of a single file. That sounds reasonable. Except, as far as I can tell, it isn't. I see no reason hcache could not significantly speed up scanning mbox folders as well, at least on any system that supports lseek() or similar (which I imagine is any system that Mutt runs on currently). The amount of benefit you'd get from this would greatly depend on the nature of the messages stored in the folder, though... Folders of moderate size or larger, with mostly large messages (or attachments) should see the most benefit, and those with many small messages, or with very few messages, would see the least (but still some). -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. pgpej_axS7hnD.pgp Description: PGP signature
Re: header_cache for mbox
While we're at it: is it at all possible (vanilla or patched) to enable hcache for _some_ mailboxes but not for others? The documentation explains that it is possible to force separate hcache database files for distinct mailboxes, so this doesn't look like a huge stretch. -- Please *no* private copies of mailing list or newsgroup messages. Rule 420: All persons more than eight miles high to leave the court.
Re: header_cache for mbox
Derek Martin wrote: On Tue, Jul 14, 2015 at 08:23:25PM +0200, Heinz Diehl wrote: On 14.07.2015, Kevin J. McCarthy wrote: With mbox, I guess the designers thought there wouldn't be that much of a speed improvement because it's just a sequential read of a single file. That sounds reasonable. Except, as far as I can tell, it isn't. I see no reason hcache could not significantly speed up scanning mbox folders as well, at least on any system that supports lseek() or similar (which I imagine is any system that Mutt runs on currently). The amount of benefit you'd get from this would greatly depend on the nature of the messages stored in the folder, though... Folders of moderate size or larger, with mostly large messages (or attachments) should see the most benefit, and those with many small messages, or with very few messages, would see the least (but still some). for lseek() to be useful, you need to know where to lseek to which you wouldn't in this case (if you want reliable parsing). and anyway, i'd like to think mutt uses mmap() for mbox files. cheers, raf
Re: header_cache for mbox
On 14Jul2015 12:36, Ian Zimmerman i...@buug.org wrote: On 2015-07-14 20:23 +0200, Heinz Diehl wrote: I'm using both IMAP and maildir, and hcache is a real and quite noticeable improvement. FWIW, I use maildir too but locally, and turning hcache on did not result in a clear speed up. My largest folders are around 5000 messages each. I use local Maildirs (and local mboxes). hcache makes an _enormous_ speedup for opening large Maildirs. I'd expect that hcache on an mbox is far less useful because if an mbox changes size or mtime you pretty much have to scan the entire thing to find out what may have changed. With POP, IMAP, Maildir you can scan for new and deleted items cheaply, allowing one to get almost everything from the cache and only do an incremental access to the Maildir or mail server. Cheers, Cameron Simpson c...@zip.com.au Footnotes that extend to a second page are an abject failure of design. - Bringhurst, _The Elements of Typographic Style_
Re: header_cache for mbox
* Eric Smith e...@fruitcom.com [07-13-15 09:17]: Has anyone implemented this or is considering to do so? Been using it for *quite* a few years now. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.orgopenSUSE Community Memberfacebook/ptilopteri http://wahoo.no-ip.orgPhoto Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535@ http://linuxcounter.net
Re: header_cache for mbox
On 13.07.2015, Patrick Shanahan wrote: * Eric Smith e...@fruitcom.com [07-13-15 09:17]: Has anyone implemented this or is considering to do so? Been using it for *quite* a few years now. AFAIK, mutt's header cache only works for IMAP and Maildir, but not for the mbox format.
Re: header_cache for mbox
* Heinz Diehl htd...@fritha.org [07-13-15 14:42]: On 13.07.2015, Patrick Shanahan wrote: * Eric Smith e...@fruitcom.com [07-13-15 09:17]: Has anyone implemented this or is considering to do so? Been using it for *quite* a few years now. AFAIK, mutt's header cache only works for IMAP and Maildir, but not for the mbox format. Guess more research is warranted. I have used mbox for 10 years and header_cache since implementation on openSUSE. user:~ mutt -v Mutt 1.5.23 (2014-03-12) Copyright (C) 1996-2009 Michael R. Elkins and others. Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'. Mutt is free software, and you are welcome to redistribute it under certain conditions; type `mutt -vv' for details. System: Linux 3.11.10-29-desktop (x86_64) ncurses: ncurses 5.9.20130805 (compiled with 5.9) libidn: 1.25 (compiled with 1.25) hcache backend: GDBM version 1.10. 13/11/2011 Compiler: Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib64/gcc/x86_64-suse-linux/4.8/lto-wrapper Target: x86_64-suse-linux Configured with: ../configure --prefix=/usr --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.8 --enable-ssp --disable-libssp --disable-plugin --with-bugurl=http://bugs.opensuse.org/ --with-pkgversion='SUSE Linux' --disable-libgcj --disable-libmudflap --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --enable-version-specific-runtime-libs --enable-linker-build-id --program-suffix=-4.8 --enable-linux-futex --without-system-libunwind --with-arch-32=i586 --with-tune=generic --build=x86_64-suse-linux Thread model: posix gcc version 4.8.1 20130909 [gcc-4_8-branch revision 202388] (SUSE Linux) Configure options: 'x86_64-suse-linux' '--prefix=/usr' '--datadir=/usr/share' '--sysconfdir=/etc' '--mandir=/usr/share/man' '--with-docdir=/usr/share/doc/packages/mutt' '--enable-imap' '--enable-pop' '--enable-pgp' '--enable-gpgme' '--enable-nfs-fix' '--enable-mailtool' '--enable-compressed' '--disable-external-dotlock' '--with-ssl=/usr' '--with-sasl=/usr' '--with-gss=/usr' '--with-gnutls=/usr' '--with-curses=/usr' '--with-regex' '--enable-smtp' '--enable-hcache' '--with-idn' 'build_alias=x86_64-suse-linux' 'host_alias=x86_64-suse-linux' 'target_alias=x86_64-suse-linux' 'CC=gcc' 'CFLAGS=-Wall -fmessage-length=0 -grecord-gcc-switches -fstack-protector -O2 -Wall -D_FORTIFY_SOURCE=2 -funwind-tables -fasynchronous-unwind-tables -g -I. -D_GNU_SOURCE -fno-strict-aliasing -fPIE -g3 -pipe' 'LDFLAGS=-Wl,--as-needed -Wl,-O2 -Wl,--hash-size=8599 -pie' Compilation CFLAGS: -Wall -pedantic -Wno-long-long -Wall -fmessage-length=0 -grecord-gcc-switches -fstack-protector -O2 -Wall -D_FORTIFY_SOURCE=2 -funwind-tables -fasynchronous-unwind-tables -g -I. -D_GNU_SOURCE -fno-strict-aliasing -fPIE -g3 -pipe Compile options: -DOMAIN -DEBUG -HOMESPOOL -USE_SETGID +USE_DOTLOCK -DL_STANDALONE +USE_FCNTL -USE_FLOCK +USE_POP +USE_IMAP +USE_SMTP +USE_SSL_OPENSSL -USE_SSL_GNUTLS +USE_SASL +USE_GSS +HAVE_GETADDRINFO -HAVE_REGCOMP +USE_GNU_REGEX +COMPRESSED +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM +CRYPT_BACKEND_CLASSIC_PGP +CRYPT_BACKEND_CLASSIC_SMIME +CRYPT_BACKEND_GPGME -EXACT_ADDRESS +SUN_ATTACHMENT +ENABLE_NLS -LOCALES_HACK +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR +HAVE_ICONV -ICONV_NONTRANS +HAVE_LIBIDN +HAVE_GETSID +USE_HCACHE ISPELL=/usr/bin/ispell SENDMAIL=/usr/sbin/sendmail MAILPATH=/var/mail PKGDATADIR=/usr/share/mutt SYSCONFDIR=/etc EXECSHELL=/bin/sh -MIXMASTER To contact the developers, please mail to mutt-...@mutt.org. To report a bug, please visit http://bugs.mutt.org/. patch-1.5.3.vk.pgp_verbose_mime patch-1.5.9.aw.listreply.1 patch-1.5.23.sidebar.20140412.txt rr.compressed But you didn't say what distro/version/compile options in your environment, so your version of mutt may indeed not support mbox header caching. Mine does. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.orgopenSUSE Community Memberfacebook/ptilopteri http://wahoo.no-ip.orgPhoto Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535@ http://linuxcounter.net
