Re: Virtual Folders in mutt
On Wed, May 22, 2002 00:19:09 at 12:19:09AM +0200, Rocco Rutte wrote: * Marco Fioretti [05/21/02 23:32:56 CEST] wrote: I too am really interested in the possibility of having all your mail indexed so that you can make faster and more sophisticated searches than grepmail allows. Couldn't the indexing database be run once and then only on *new* messages, after fetchmail/procmail have delivered them? This should not be noticeable, should it? You have to keep it outside procmail to make it more flexible (also, procmail itself is already too slow). Depending on what exactly you want to index, What I know is that I **badly** need a fast way to type some keywords/parts of sentences, and find all the relevant files on my drive *and* email messages where I or somebody else already wrote about it. It should be useable from the command line, from mutt, and from Emacs too. I have discovered just three days ago the existance of the Remembrance agent, but haven't yet downloaded, or even figured out if it *is* the right/fastest way to solve my problem. Any help is appreciated. Running it by procmail on any message would sure be slow, but I was thinking to some script starting *after* procmail, and parsing all the messages in all folders not scanned since the previous running. This would cover even the messages never read. What about the indexer (whatever it is) that, if the text being parsed is an email message: add an header to messages with some numeric code/hash to make *its* later searches faster, can be launched later from mutt so that it converts the keywords in some other numeric code hash to be easily checked against that header, giving a much faster result? DISCLAIMER: I still know nothing of indexing software (otherwise I'd be founding nextgoogle.com :-) ), and it's six am. If the above suggestion is sheer idiocy, kindly point me to some primer on the matter, thanks! Ciao, Marco Fioretti -- Computers are useless. They can only give you answers. -- Pablo Picasso
Re: Virtual Folders in mutt
On Mon, May 20, 2002 at 09:36:04AM +0100, Bruno Postle wrote: On Mon 20-May-2002 at 05:50:38 +1000, Iain Truskett wrote: * Anthony Towns ([EMAIL PROTECTED]) [20 May 2002 17:38]: First, has anyone done this before? Is there a FAQ or HOWTO I could be reading? Examine the 'limit' command (bound by default to 'l' in the index screen). The limit command is perfect for this, but it only works on one mailbox and mailboxes don't scale too well - Ok for five thousand messages, _not_ so ok for a million. It's not that okay with five thousand messages, and it takes minutes to open up a folder that's something like half my archive. It's not really the way I'd rather work either: I'd rather tell it what I want, wait if necessary, then see the messages, rather than wait while it shows me all the messages I might want, then tell it what I'm looking for. Limit's nice, but it's not what I'm looking for. On Mon, May 20, 2002 at 11:47:59AM -0400, Adam Shostack wrote: On Mon, May 20, 2002 at 06:39:58AM -0500, Patrick wrote: Look at grepmail, grepmail.sourceforge.net. Grepmail doesn't seem to support Maildir, which I'm inclined to stick with. I'm really more interested in making something-like-grepmail more comfortable to use with mutt, too: I'm pretty sure I can write something-like-grepmail myself to my satisfaction. One important difference is that vfolders are built around pre-built indexes, making them more efficient than grepping hundreds of megs of mail. Having watched people use vfolders, it makes me seriously consider switching over to evolution. Right. The main thing is that as well as the straight out mbox/Maildir, there's a nice binary index of all the messages that's quick to look up. However, I can't find a decent technical overview of the system, other than the source. Likewise. file(1) couldn't work out what format the index was in either, so I gave up. :) Bruno Postle pointed out some similar stuff for mutt, though. Apparently Michael Elkins has a patch that adds header caching support for Maildir folders, which is a start. It's at: http://www.cs.hmc.edu/~me/mutt/patch-1.5.0.me.hcache.8 Bruno's posted a grep script which makes use of this stuff too (back in January), which is at: http://groups.yahoo.com/group/mutt-users/message/27023 On Mon, May 20, 2002 at 08:23:11AM -0500, David T-G wrote: % thinking that vFolders might be the answer. So, I've been trying to figure % out some way of doing the same sort of thing in mutt. Interesting concept... Since I have no interest in installing Evo, can you point me somewhere to learn more about their concept and implementation? The concept is basically the same as the limit stuff, although the interface allows you to make a mailbox that's a vfolder -- so you can just change to it rather than having to change to the big folder, then limit it by hand. You can give them names and such too, which is nice if you do this sort of thing regularly, and makes it worthwhile to have queries like All unread messages to mailing lists in the past hour, if that's what you're into. There arn't any technical docs about the implementation afaik, but it's basically make all accesses to the mail really fast by using indexes, then have at it. It's not clear what the indexes are, but I figure I can fake that with some scripts well enough for my own purposes. Apparently one of the nicer ways to use it is for incoming mail, but that's probably a lot tougher to adapt mutt for, and not really what I'm interested in, personally. I do know, though, that I've been itching for a real RDBMS back-end for mailboxes [...] I want to keep mine in the filesystem: my other annoyance is that I want to be able to handle my mail without any stress on either of two computers which aren't always connected to each other. I've tried this before, and I tend to have problems when I read some mail on one machine, and archive it; then read it on the other and archive it again. I think Maildir will be a part of my solution when I find one though. I had been tinkering with an idea for approaching that by having one single, potentially huge, canonical Maildir containing every message, and then virtual Maildirs (which, BTW, need only the cur/ subdir) with symlinks that point into the canonical dir. Right, this is more or less what I'm planning on doing, except that my virtual Maildirs will be created on demand, and cleaned up when they're not used. Also, I found mutt just ignored symlinks, so I used hardlinks instead. % What I can't figure out is how to run that script and make mutt look at % the output in a reasonably effective way. What I'd _like_ is to be able % to write a macro that asks me: % What email address: % then runs change-folder`/home/aj/mail/bin/query-by-addr $result`. Why not just fire off a subprocess? Because then I'll end up with fifty mutts one on top of the other as I refine my query, which will annoy me to no end,
Re: Virtual Folders in mutt
Hi, At 3:32 AM EDT on May 20 Anthony Towns sent off: I've been getting gradually more annoyed with the way my mail archives are organised recently, and, after playing with Evolution a little bit, got to thinking that vFolders might be the answer. So, I've been trying to figure out some way of doing the same sort of thing in mutt. First, has anyone done this before? Is there a FAQ or HOWTO I could be reading? What I'm thinking is basically having a huge morass of mail shoved in a directory somewhere, with a fancy index. Whenever I want to look at mail in there, I construct a query (Show me all mail with [EMAIL PROTECTED] in the From/To/Cc headers or Show me all mail with message-id blah) and then have mutt show me all the mail that matches that query as a virtual folder. This isn't exactly what you asked for, but you might want to look at the Remembrance Agent, http://rhodes.www.media.mit.edu/people/rhodes/RA/ -- We have to pursue this subject of fun very seriously if we want to stay competitive in the 21st century. - George Yeo, Singapore's Minister of State for Finance. Robert I. Reid | PGP/GPG Keys: http://astro.utoronto.ca/~reid/pgp.html
Re: Virtual Folders in mutt
On Mon, May 20, 2002 19:04:20 at 07:04:20PM +0100, Bruno Postle wrote: On Mon 20-May-2002 at 12:40:31PM -0500, David T-G wrote: % One important difference is that vfolders are built around % pre-built indexes, making them more efficient than grepping hundreds % of megs of Hmmm... Maybe glimpse as a search indexer? The Maildir 'me.hcache' patch indexes your mail as you go along and the db files are quite ok for searching. Indexing mail in this way is much more efficient than an indexing cronjob (always out-of-date) or a separate indexing demon (wrecks quake performance). What do you mean by indexing daemon? I too am really interested in the possibility of having all your mail indexed so that you can make faster and more sophisticated searches than grepmail allows. Couldn't the indexing database be run once and then only on *new* messages, after fetchmail/procmail have delivered them? This should not be noticeable, should it? The indexing database engine also has the enormous advantage that you can index every text file, not just email. Unlike the other solutions proposed, it would also help when you say damn, I should really paste into this message the last part of that old school project... What do you think? Marco Fioretti Red Hat 7.2 in 8 MB of RAM: www.rule-project.org -- Language shapes the way we think, and determines what we can think about. B. L. Whorf
Re: Virtual Folders in mutt
Hi, * Marco Fioretti [05/21/02 23:32:56 CEST] wrote: I too am really interested in the possibility of having all your mail indexed so that you can make faster and more sophisticated searches than grepmail allows. Couldn't the indexing database be run once and then only on *new* messages, after fetchmail/procmail have delivered them? This should not be noticeable, should it? You have to keep it outside procmail to make it more flexible (also, procmail itself is already too slow). Depending on what exactly you want to index, I think you'll notice it. The indexing database engine also has the enormous advantage that you can index every text file, not just email. You can also on-demand translate between the mail and the text file index. This way it would be easier to integrate a mail index into every other database rather than converting the existing database (allthough it maybe slows down search). Unlike the other solutions proposed, it would also help when you say damn, I should really paste into this message the last part of that old school project... No problem with an extendable editor. Cheers, Rocco.
Re: Virtual Folders in mutt
On Tue 21-May-2002 at 11:32:56PM +0200, Marco Fioretti wrote: The Maildir 'me.hcache' patch indexes your mail as you go along and the db files are quite ok for searching. Indexing mail in this way is much more efficient than an indexing cronjob (always out-of-date) or a separate indexing demon (wrecks quake performance). What do you mean by indexing daemon? An indexer can run as a separate background process, trawling the filesystem for new data. Microsoft 'find fast', Nautilus and other similar beasts do this (I think, apologies if I'm slandering Nautilus here). Couldn't the indexing database be run once and then only on *new* messages, after fetchmail/procmail have delivered them? This should not be noticeable, should it? That would be a good strategy, so long as procmail is the only local delivery agent. You would still need to keep track of messages you create or move with mutt though. The only disadvantage with letting mutt build the index, is that you won't be able to find messages in folders that haven't been 'opened'. -- Bruno
Virtual Folders in mutt
Hi, I've been getting gradually more annoyed with the way my mail archives are organised recently, and, after playing with Evolution a little bit, got to thinking that vFolders might be the answer. So, I've been trying to figure out some way of doing the same sort of thing in mutt. First, has anyone done this before? Is there a FAQ or HOWTO I could be reading? What I'm thinking is basically having a huge morass of mail shoved in a directory somewhere, with a fancy index. Whenever I want to look at mail in there, I construct a query (Show me all mail with [EMAIL PROTECTED] in the From/To/Cc headers or Show me all mail with message-id blah) and then have mutt show me all the mail that matches that query as a virtual folder. The easiest way to do this seems like it'd be to write a script which does the query and constructs a mailbox in Maildir format with all the appropriate messages in it, then have mutt look at that Maildir. That much I can do. What I can't figure out is how to run that script and make mutt look at the output in a reasonably effective way. What I'd _like_ is to be able to write a macro that asks me: What email address: then runs change-folder`/home/aj/mail/bin/query-by-addr $result`. (I'd like to be able to have a few mutts doing queries at the same time, so the folder name needs to be determined somewhat dynamically to avoid clashes. I can't figure out how to do that with a macro either) I could avoid the prompting by having my script do that, but I can't figure out any way to have a macro change to a dynamically-determined folder. What would be ideal would be to have some sort of user variable that could be set, and some way of making a macro give a prompt then set one of those user variables. Then I could say: macro index Q[What email address:,email] \ :set user_foldername=`query-by-addr $email`enter \ change-folder$foldernameenter or something similar. But I can't do that, and I'm stumped as to what else I can do. Any ideas? Cheers, aj -- Anthony Towns [EMAIL PROTECTED] http://azure.humbug.org.au/~aj/ I don't speak for anyone save myself. GPG signed mail preferred. ``BAM! Science triumphs again!'' -- http://www.angryflower.com/vegeta.gif msg28213/pgp0.pgp Description: PGP signature
Re: Virtual Folders in mutt
* Anthony Towns ([EMAIL PROTECTED]) [20 May 2002 17:38]: [...] First, has anyone done this before? Is there a FAQ or HOWTO I could be reading? Examine the 'limit' command (bound by default to 'l' in the index screen). [...] What I can't figure out is how to run that script and make mutt look at the output in a reasonably effective way. What I'd _like_ is to be able to write a macro that asks me: What email address: Enter mailbox in usual fashion l ~f [EMAIL PROTECTED] That's how I do things at least. cheers, -- iain. http://eh.org/~koschei/ msg28214/pgp0.pgp Description: PGP signature
Re: Virtual Folders in mutt
On Mon 20-May-2002 at 05:50:38 +1000, Iain Truskett wrote: * Anthony Towns ([EMAIL PROTECTED]) [20 May 2002 17:38]: First, has anyone done this before? Is there a FAQ or HOWTO I could be reading? Examine the 'limit' command (bound by default to 'l' in the index screen). The limit command is perfect for this, but it only works on one mailbox and mailboxes don't scale too well - Ok for five thousand messages, _not_ so ok for a million. -- Bruno
Re: Virtual Folders in mutt
* Bruno Postle [EMAIL PROTECTED] [05-20-02 03:37]: On Mon 20-May-2002 at 05:50:38 +1000, Iain Truskett wrote: * Anthony Towns ([EMAIL PROTECTED]) [20 May 2002 17:38]: First, has anyone done this before? Is there a FAQ or HOWTO I could be reading? Examine the 'limit' command (bound by default to 'l' in the index screen). The limit command is perfect for this, but it only works on one mailbox and mailboxes don't scale too well - Ok for five thousand messages, _not_ so ok for a million. Look at grepmail, grepmail.sourceforge.net. -- Patrick Shanahan Registered Linux User #207535 @ http://counter.li.org
Re: Virtual Folders in mutt
Anthony, et al -- ...and then Anthony Towns said... % % Hi, Hello! % % I've been getting gradually more annoyed with the way my mail archives are % organised recently, and, after playing with Evolution a little bit, got to *grin* % thinking that vFolders might be the answer. So, I've been trying to figure % out some way of doing the same sort of thing in mutt. Interesting concept... Since I have no interest in installing Evo, can you point me somewhere to learn more about their concept and implementation? % % First, has anyone done this before? Is there a FAQ or HOWTO I could % be reading? Can't properly help you there until I really know what you mean :-) I do know, though, that I've been itching for a real RDBMS back-end for mailboxes so that I can simultaneously get around the where do I put this? problem (eg, an offshoot mail to Bob really belongs in Alice's thread and therefore her mailbox) and the how do I avoid duplication? problem (sure, I could [somehow, as has been discussed in fcc-save-hook threads before] save it in Bob's and Alice's folders, but that's wasteful and it still doesn't point to Alice's from Bob's), but that's a long way off. I had been tinkering with an idea for approaching that by having one single, potentially huge, canonical Maildir containing every message, and then virtual Maildirs (which, BTW, need only the cur/ subdir) with symlinks that point into the canonical dir. The tricks are 1) organizing the fake mailboxes and keeping them updated (procmail-ish) and saving mail in them by actually writing symlinks and saving to the big one (mutt-ish). It's only a workaround anyway, so it probably doesn't deserve much implementation effort, but it can stand some though to look at ways to develop this. % % What I'm thinking is basically having a huge morass of mail shoved in a % directory somewhere, with a fancy index. Whenever I want to look at mail Exactly. % in there, I construct a query (Show me all mail with [EMAIL PROTECTED] % in the From/To/Cc headers or Show me all mail with message-id blah) % and then have mutt show me all the mail that matches that query as a % virtual folder. You can probably construct such a query with grepm (a mutt-driving frontend for grepmail, as noted in another reply), but the trick is the search time involved (and the fact that grepmail works on mbox files specifically because they are trickier to parse at the shell level) and typing out the command (probably easier with macros or a renamed script). % % The easiest way to do this seems like it'd be to write a script which % does the query and constructs a mailbox in Maildir format with all the % appropriate messages in it, then have mutt look at that Maildir. That % much I can do. Again, see grepmail and grepm here. % % What I can't figure out is how to run that script and make mutt look at % the output in a reasonably effective way. What I'd _like_ is to be able % to write a macro that asks me: % % What email address: % % then runs change-folder`/home/aj/mail/bin/query-by-addr $result`. Why not just fire off a subprocess? % % (I'd like to be able to have a few mutts doing queries at the same time, % so the folder name needs to be determined somewhat dynamically to avoid % clashes. I can't figure out how to do that with a macro either) % % I could avoid the prompting by having my script do that, but I can't % figure out any way to have a macro change to a dynamically-determined % folder. Looks OK if you change the implementation. % % What would be ideal would be to have some sort of user variable that % could be set, and some way of making a macro give a prompt then set one % of those user variables. Then I could say: % % macro index Q[What email address:,email] \ % :set user_foldername=`query-by-addr $email`enter \ % change-folder$foldernameenter You're asking for a scripting language; that's been requested before, too :-) % % or something similar. But I can't do that, and I'm stumped as to what else % I can do. % % Any ideas? I'm quite interested in my two ideas above, but I'd also be interested in a quick indexing system for Maildir and a way for mutt, this indexing system, and a search and assembly tool to work together on compressed mailboxes. % % Cheers, % aj % % -- % Anthony Towns [EMAIL PROTECTED] http://azure.humbug.org.au/~aj/ % I don't speak for anyone save myself. GPG signed mail preferred. % % ``BAM! Science triumphs again!'' % -- http://www.angryflower.com/vegeta.gif TIA HAND Good luck! :-D -- David T-G * It's easier to fight for one's principles (play) [EMAIL PROTECTED] * than to live up to them. -- fortune cookie (work) [EMAIL PROTECTED] http://www.justpickone.org/davidtg/Shpx gur Pbzzhavpngvbaf Qrprapl Npg! msg28221/pgp0.pgp Description: PGP signature
Re: Virtual Folders in mutt
On Mon, May 20, 2002 at 06:39:58AM -0500, Patrick wrote: * Bruno Postle [EMAIL PROTECTED] [05-20-02 03:37]: On Mon 20-May-2002 at 05:50:38 +1000, Iain Truskett wrote: * Anthony Towns ([EMAIL PROTECTED]) [20 May 2002 17:38]: First, has anyone done this before? Is there a FAQ or HOWTO I could be reading? Examine the 'limit' command (bound by default to 'l' in the index screen). The limit command is perfect for this, but it only works on one mailbox and mailboxes don't scale too well - Ok for five thousand messages, _not_ so ok for a million. Look at grepmail, grepmail.sourceforge.net. One important difference is that vfolders are built around pre-built indexes, making them more efficient than grepping hundreds of megs of mail. Having watched people use vfolders, it makes me seriously consider switching over to evolution. However, I can't find a decent technical overview of the system, other than the source. Adam
Re: Virtual Folders in mutt
On Mon 20-May-2002 at 12:40:31PM -0500, David T-G wrote: % One important difference is that vfolders are built around % pre-built indexes, making them more efficient than grepping hundreds % of megs of Hmmm... Maybe glimpse as a search indexer? The Maildir 'me.hcache' patch indexes your mail as you go along and the db files are quite ok for searching. Indexing mail in this way is much more efficient than an indexing cronjob (always out-of-date) or a separate indexing demon (wrecks quake performance). -- Bruno
Re: Virtual Folders in mutt
On Mon, May 20, 2002 at 12:40:31PM -0500, David T-G wrote: Adam, et al -- ...and then Adam Shostack said... % % On Mon, May 20, 2002 at 06:39:58AM -0500, Patrick wrote: ... % _not_ so ok for a million. % % Look at grepmail, grepmail.sourceforge.net. % % One important difference is that vfolders are built around pre-built % indexes, making them more efficient than grepping hundreds of megs of Hmmm... Maybe glimpse as a search indexer? Mailing list archive search engines are fast; can the same idea be used to return a list of messages from lots of mailboxes being indexed and to hand to mutt as a mailbox? Yes, I used to have this, but it never got today's mail, and it wasn't really integrated into anything. If I wanted non-integrated mail, I'd be using MH. :) % mail. Having watched people use vfolders, it makes me seriously % consider switching over to evolution. Heavens! :-) I figured that would be attention grabbing. :) % % However, I can't find a decent technical overview of the system, other % than the source. Is there even a decent non-technical interview, or perhaps a non-decent technical interview? What goal are they accomplishing and, very basically, how is it done? http://www.ximian.com/devzone/tech/evolution.html http://www.ximian.com/devzone/manuals/evolution/usage-mail-organize-vfolders.html Adam
