Re: OT: Managing huge Mail/ folders (with mutt?)
James Sinnamon wrote: Thanks everyone for all your ideas. My previously unmanageable Mail directory will be manageable again very soon, but right now I will be putting it on hold until I get my own Mailman server working. Some further questions (feel welcome to post any response to debian-user): On Mon, 12 Jul 2004 09:17 pm, John Summerfield wrote: Better, turn on Kmail's expiration processing. Where do I find this? What sub-menu? Does it allow me the option of putting e-mails into an archive, or does it just purge them? Choose a folder Click Folder on the menu bar, choose properties. Read the form. I think it gets plonked. You can also get the dialogue by right-clicking the folder. Somewhere else, you get to choose _when_ it expires. Probably you don't want it asking silly questions all the time, or having to wait an hour or two:-) -- Cheers John -- spambait [EMAIL PROTECTED] [EMAIL PROTECTED] Tourist pics http://portgeographe.environmentaldisasters.cds.merseine.nu/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT: Managing huge Mail/ folders (with mutt?)
Incoming from Karsten M. Self: :0 * (^TO|^X-Mailing-List:.*)debian { :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-openoffice/ With a little care, you might replace all of the latter with one recipe: :0: * ^X-Mailing-List:.*debian-\/@lists\.debian\.org $LISTDIR/Debian/debian-${MATCH}/ -- Any technology distinguishable from magic is insufficiently advanced. (*) http://www.spots.ab.ca/~keeling - - -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT: Managing huge Mail/ folders (with mutt?)
Incoming from James Sinnamon: Karsten and others, Firstly, thank you all for the responses. On Mon, 12 Jul 2004 02:35 pm, Karsten M. Self wrote: on Sun, Jul 11, 2004 at 07:18:55PM +1000, James Sinnamon snip/ So could anyone tell me how they handle ever growing Mail folders? Perhaps 'mutt' is the way to go? Procmail, or its equivalents. snip/ On Sun, 11 Jul 2004 10:32 pm, S.D.A. wrote: What I do, is use a nice little utility called archivemail and set it up I take it that archivemail can handle mail folders and files which are I don't know what he uses for archiving mail, but a shell script wrapped around gzip that appends mail to a gzip archive works well. cron runs mine: -- PATH=/usr/bin:/bin MAILDIR=/home/keeling/Mail YEAR=$(date '+%Y') for f in abuse headhunters headhunter_reply procmail root spamcop-reports spam; do if [ -e ${MAILDIR}/${f} ]; then cat ${MAILDIR}/${f} | gzip /home/keeling/dox/archive_Mail/${YEAR}_Mail_${f}.gz sleep 2 sync ; sync ; sync echo ${0}: archived ${f} to: \$HOME/dox/archive_Mail/${YEAR}_Mail_${f}.gz cat /dev/null ${MAILDIR}/${f} fi done -- If your MUA (mutt?) is smart enough, it can even read gzipped mail folders. -- Any technology distinguishable from magic is insufficiently advanced. (*) http://www.spots.ab.ca/~keeling - - -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT: Managing huge Mail/ folders (with mutt?)
on Sun, Jul 11, 2004 at 07:18:55PM +1000, James Sinnamon ([EMAIL PROTECTED]) wrote: Dear List, My ${HOME}/Mail directory is currently nearly 350 Megs in size. A lot of it is due to high volume mailing lists such as debian-user (48 Meg so far), and this can only get much worse as I join more and more high volume lists. So could anyone tell me how they handle ever growing Mail folders? Perhaps 'mutt' is the way to go? I had thought of splitting the {HOME}/Mail into two and run a second instance of Kmail, with a different profile, to handle the mailing lists, but KMail doesn't seem to allow for this as far as I can tell (perhaps for some good reason). What you want is procmail (or a similar utility). There's a long history of tools in 'Nix for managing email. Filtering stuff to mailboxes by various criteria is only one side of it. My own procmail recipies: - Add some useful headers to mail - Filter list mail to folders. - Deal with some high-priority mail (goes straight to my inbox). - Keeps track of senders who are on white or grey lists (or a couple of others). - Handles spam with the addition of spamassassin. You can do all sorts of stuff, including forwarding mail, executing programs, and whatnot. The syntax is slighly (ok, very) arcane, but powerful. There's a good set of prepared recipies in the spamtools package, based on Lars Wirzenius's procmail filters. Do *not* enable the autoreply feature, but learn from the rest of it. My own filters run over 24 files and 2000 lines, though most of it I don't have to mess with (actually, splitting it into that many files makes it easier to handle). A small portion of my list management rules follows. It handles my Debian list mail. The first rule identifies the lists globally, the subsequent rules apply *only* to mail meeting the first, and drops mail into the appropriate folder. :0 * (^TO|^X-Mailing-List:.*)debian { :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-openoffice/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-qa/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-private/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-changes/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-policy/ :0: * ^X-Mailing-List:.*debian-(admintool|devel)@lists.debian.org $LISTDIR/Debian/debian-devel/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-user/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-bugs/ # Debian Laptop :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-laptop/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-security/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-pilot/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-devel/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-firewall/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-python/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-testing/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-bugs/ :0: * 1^0 (^TO|^X-Mailing-List:.*)[EMAIL PROTECTED] * 1^0 (^TO|^X-Mailing-List:.*)[EMAIL PROTECTED] $INBOX # Catch-all :0: $INBOX } Peace. -- Karsten M. Self [EMAIL PROTECTED]http://kmself.home.netcom.com/ What Part of Gestalt don't you understand? The opposite of increment is excrement. - Seen on Usenet, so it must be true. signature.asc Description: Digital signature
Re: OT: Managing huge Mail/ folders (with mutt?)
Karsten M. Self wrote: on Sun, Jul 11, 2004 at 07:18:55PM +1000, James Sinnamon ([EMAIL PROTECTED]) wrote: Dear List, My ${HOME}/Mail directory is currently nearly 350 Megs in size. A lot of it is due to high volume mailing lists such as debian-user (48 Meg so far), and this can only get much worse as I join more and more high volume lists. So could anyone tell me how they handle ever growing Mail folders? Perhaps 'mutt' is the way to go? I had thought of splitting the {HOME}/Mail into two and run a second instance of Kmail, with a different profile, to handle the mailing lists, but KMail doesn't seem to allow for this as far as I can tell (perhaps for some good reason). What you want is procmail (or a similar utility). Better, turn on Kmail's expiration processing. -- Cheers John -- spambait [EMAIL PROTECTED] [EMAIL PROTECTED] Tourist pics http://portgeographe.environmentaldisasters.cds.merseine.nu/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT: Managing huge Mail/ folders (with mutt?)
on Mon, Jul 12, 2004 at 12:05:10AM -0600, s. keeling ([EMAIL PROTECTED]) wrote: Incoming from Karsten M. Self: :0 * (^TO|^X-Mailing-List:.*)debian { :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-openoffice/ With a little care, you might replace all of the latter with one recipe: :0: * ^X-Mailing-List:.*debian-\/@lists\.debian\.org $LISTDIR/Debian/debian-${MATCH}/ Nick Moffitt's posted a similar recipie which matches on pretty much all the common (and otherwise) list serv software headers, and automagickally filters the mail accordingly. Pretty slick, all told. But I drive stick. Peace. -- Karsten M. Self [EMAIL PROTECTED]http://kmself.home.netcom.com/ What Part of Gestalt don't you understand? Moderator, Free Software Law Discussion mailing list: http://lists.alt.org/mailman/listinfo/fsl-discuss/ signature.asc Description: Digital signature
Re: OT: Managing huge Mail/ folders (with mutt?)
on Mon, Jul 12, 2004 at 03:49:25PM +1000, James Sinnamon ([EMAIL PROTECTED]) wrote: Karsten and others, Firstly, thank you all for the responses. NP. On Mon, 12 Jul 2004 02:35 pm, Karsten M. Self wrote: on Sun, Jul 11, 2004 at 07:18:55PM +1000, James Sinnamon snip/ So could anyone tell me how they handle ever growing Mail folders? Perhaps 'mutt' is the way to go? Procmail, or its equivalents. ... Of course you would be aware that Kmail's filtering capabilities can do something similar to what is done above. Do you use procmail instead of, or as a complement to, a GUI e-mail client such as Kmail? The difference is this: - If you use KMail's filtering tools, when you decide to switch to another mail client (permanently or temporarially), you lose the filtering. - If you use procmail, your filters are independent of your mailer. I can access my mail with mutt, balsa, kmail, evolution, or the shell. Procmail doesn't care. This is the advantage of proper scoping and modularization of tools. It's a powerful concept. I take it that archivemail can handle mail folders and files which are already broken down in to sub-folders, such as mine, some of which are shown below? Procmail reads messages from stdin. Generally as the mail is delivered. So that when you open your mail client, the mail is filtered and waiting for you. You can also (re)filter existing mailboxes by various means. Peace. -- Karsten M. Self [EMAIL PROTECTED]http://kmself.home.netcom.com/ What Part of Gestalt don't you understand? GNOME (and real UI developers) READ THIS: http://use.perl.org/~btilly/journal/18678 signature.asc Description: Digital signature
Re: OT: Managing huge Mail/ folders (with mutt?)
Karsten M. Self wrote: on Mon, Jul 12, 2004 at 03:49:25PM +1000, James Sinnamon ([EMAIL PROTECTED]) wrote: Karsten and others, Firstly, thank you all for the responses. NP. On Mon, 12 Jul 2004 02:35 pm, Karsten M. Self wrote: on Sun, Jul 11, 2004 at 07:18:55PM +1000, James Sinnamon snip/ So could anyone tell me how they handle ever growing Mail folders? Perhaps 'mutt' is the way to go? Procmail, or its equivalents. ... Of course you would be aware that Kmail's filtering capabilities can do something similar to what is done above. Do you use procmail instead of, or as a complement to, a GUI e-mail client such as Kmail? The difference is this: - If you use KMail's filtering tools, when you decide to switch to another mail client (permanently or temporarially), you lose the filtering. It's a standard kmail feature. Most people do not want to come to terms with procmail: I've used it, I had some pretty fancy filters that could spot most mailing lists, even those I'd never seen before, and drop the email into the appropriate folder, creating it if necessary. Of course, if I changed email client I'd have to change those filters too, because the email folder format would likely be different. Now I have an IMAP email service where per-user procmail filters are not possible. At the time I was using procmail, using it with kmail was unsafe - the authors said so. - If you use procmail, your filters are independent of your mailer. I can access my mail with mutt, balsa, kmail, evolution, or the shell. Procmail doesn't care. This is the advantage of proper scoping and modularization of tools. It's a powerful concept. Procmail is fine for handling email at the time of delivery. In this case, the email's already in the folders.Kmail folders. Kmail can expire old email, all by itself. -- Cheers John -- spambait [EMAIL PROTECTED] [EMAIL PROTECTED] Tourist pics http://portgeographe.environmentaldisasters.cds.merseine.nu/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT: Managing huge Mail/ folders (with mutt?)
On Sun, Jul 11, 2004 at 07:18:55PM +1000 or thereabouts, James Sinnamon wrote: Dear List, My ${HOME}/Mail directory is currently nearly 350 Megs in size. A lot of it is due to high volume mailing lists such as debian-user (48 Meg so far), and this can only get much worse as I join more and more high volume lists. So could anyone tell me how they handle ever growing Mail folders? Perhaps 'mutt' is the way to go? I had thought of splitting the {HOME}/Mail into two and run a second instance of Kmail, with a different profile, to handle the mailing lists, but KMail doesn't seem to allow for this as far as I can tell (perhaps for some good reason). What I do, is use a nice little utility called archivemail and set it up via CRON to archive my mail folders (mbox) at least once a month. Then I use another utility called grepmail which can search both my active mboxes and the compressed archived mail. -- Steve + Sunday Jul 11 2004 08:21:02 AM EDT + The human mind treats a new idea the way the body treats a strange protein -- it rejects it. -- P. Medawar pgpBTZLKZLIuz.pgp Description: PGP signature
Re: OT: Managing huge Mail/ folders (with mutt?)
Incoming from S.D.A.: On Sun, Jul 11, 2004 at 07:18:55PM +1000 or thereabouts, James Sinnamon wrote: My ${HOME}/Mail directory is currently nearly 350 Megs in size. A lot of it is due to high volume mailing lists such as debian-user (48 Meg so far), and this can only get much worse as I join more and more high volume lists. So could anyone tell me how they handle ever growing Mail folders? Perhaps 'mutt' is the way to go? Though I use and recommend mutt, it has little to do with this problem. What I do, is use a nice little utility called archivemail and set it up via CRON to archive my mail folders (mbox) at least once a month. Then I use another utility called grepmail which can search both my active mboxes and the compressed archived mail. I do all that too. However, for the OP, it might be simpler to just not archive mailing list mail. You're duplicating lists.debian.org. If it's all out there in a searchable archive, why do you also need it on your box? -- Any technology distinguishable from magic is insufficiently advanced. (*) http://www.spots.ab.ca/~keeling - - -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT: Managing huge Mail/ folders (with mutt?)
On Sun, Jul 11, 2004 at 09:23:20AM -0600 or thereabouts, s. keeling wrote: Incoming from S.D.A.: On Sun, Jul 11, 2004 at 07:18:55PM +1000 or thereabouts, James Sinnamon wrote: snip So could anyone tell me how they handle ever growing Mail folders? Perhaps 'mutt' is the way to go? Though I use and recommend mutt, it has little to do with this problem. What I do, is use a nice little utility called archivemail and set it up via CRON to archive my mail folders (mbox) at least once a month. Then I use another utility called grepmail which can search both my active mboxes and the compressed archived mail. I do all that too. However, for the OP, it might be simpler to just not archive mailing list mail. You're duplicating lists.debian.org. If it's all out there in a searchable archive, why do you also need it on your box? Good point. Personally, I prefer to have an archive locally as opposed to online, for those occasional times, one can't reach the 'net. -- Steve + Sunday Jul 11 2004 12:26:01 PM EDT + pgp1sjNAWyeCA.pgp Description: PGP signature
Re: OT: Managing huge Mail/ folders (with mutt?)
On Sun, Jul 11, 2004 at 09:23:20AM -0600, s. keeling wrote: ... I do all that too. However, for the OP, it might be simpler to just not archive mailing list mail. You're duplicating lists.debian.org. If it's all out there in a searchable archive, why do you also need it on your box? The letters I hold on to typically have some info I figure I may want/need later. I could see holding those locally since searching them can be much easier/faster than anything online (including Google). If the OP holds ALL the list mail, then I see that as a waste of space, and agree with your assessment. Time for me to look at those tools... heheh. Kenward -- In a completely rational society, the best of us would aspire to be _teachers_ and the rest of us would have to settle for something less, because passing civilization along from one generation to the next ought to be the highest honor and the highest responsibility anyone could have. - Lee Iacocca -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT: Managing huge Mail/ folders (with mutt?)
on Sun, Jul 11, 2004 at 07:18:55PM +1000, James Sinnamon ([EMAIL PROTECTED]) wrote: Dear List, My ${HOME}/Mail directory is currently nearly 350 Megs in size. A lot of it is due to high volume mailing lists such as debian-user (48 Meg so far), and this can only get much worse as I join more and more high volume lists. So could anyone tell me how they handle ever growing Mail folders? Perhaps 'mutt' is the way to go? Procmail, or its equivalents. There's a long, long tradition of tools in 'Nix to automate filtering (and other processing) of email. Procmail is one of the traditional tools for doing this. You can even start with some largely configured scripts by installing the spamfilter package (based on Lars Wirzenius's procmail recipies). At its simplest, procmail allows you to direct mail into appropriate buckets by mailing list. I use mine to additionally: - Filter duplicate messages - Filter spam - Filter out blacklisted addresses/domains - Add various headers to mail It runs to 24 files and 2700+ lines, most of which is largely stock. Sounds intimidating, but works pretty damned well, and rarely needs tweaking. Primary exeptions are the list rules as I subscribe to new mailing lists (and there are ways to largely automate this, if you trust headers). A small section of my own list.rules file, handling Debian mailing lists, looks like the following. Basically, it says if this was a Debian list mail..., then tests for various list signatures and drops the mail into an appropriate box: * (^TO)[EMAIL PROTECTED] $LISTDIR/Debian/debian-bad/ :0 * (^TO|^X-Mailing-List:.*)debian { :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-openoffice/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-qa/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-private/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-changes/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-policy/ :0: * ^X-Mailing-List:.*debian-(admintool|devel)@lists.debian.org $LISTDIR/Debian/debian-devel/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-user/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-bugs/ # Debian Laptop :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-laptop/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-security/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-pilot/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-devel/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-firewall/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-python/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-testing/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-bugs/ :0: * 1^0 (^TO|^X-Mailing-List:.*)[EMAIL PROTECTED] * 1^0 (^TO|^X-Mailing-List:.*)[EMAIL PROTECTED] $INBOX # Catch-all :0: $INBOX } Peace. -- Karsten M. Self [EMAIL PROTECTED]http://kmself.home.netcom.com/ What Part of Gestalt don't you understand? GNOME (and real UI developers) READ THIS: http://use.perl.org/~btilly/journal/18678 signature.asc Description: Digital signature
Re: OT: Managing huge Mail/ folders (with mutt?)
Karsten and others, Firstly, thank you all for the responses. On Mon, 12 Jul 2004 02:35 pm, Karsten M. Self wrote: on Sun, Jul 11, 2004 at 07:18:55PM +1000, James Sinnamon snip/ So could anyone tell me how they handle ever growing Mail folders? Perhaps 'mutt' is the way to go? Procmail, or its equivalents. snip/ A small section of my own list.rules file, handling Debian mailing lists, looks like the following. Basically, it says if this was a Debian list mail..., then tests for various list signatures and drops the mail into an appropriate box: * (^TO)[EMAIL PROTECTED] $LISTDIR/Debian/debian-bad/ :0 * (^TO|^X-Mailing-List:.*)debian { :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-openoffice/ :0: * ^X-Mailing-List:[EMAIL PROTECTED] $LISTDIR/Debian/debian-qa/ snip/ :0: * 1^0 (^TO|^X-Mailing-List:.*)[EMAIL PROTECTED] * 1^0 (^TO|^X-Mailing-List:.*)[EMAIL PROTECTED] $INBOX # Catch-all :0: $INBOX } Thanks for the explanation of procmail and the example. Of course you would be aware that Kmail's filtering capabilities can do something similar to what is done above. Do you use procmail instead of, or as a complement to, a GUI e-mail client such as Kmail? On Sun, 11 Jul 2004 10:32 pm, S.D.A. wrote: What I do, is use a nice little utility called archivemail and set it up via CRON to archive my mail folders (mbox) at least once a month. Then I use another utility called grepmail which can search both my active mboxes and the compressed archived mail. I take it that archivemail can handle mail folders and files which are already broken down in to sub-folders, such as mine, some of which are shown below? --- me:Mail$ find . . ./inbox ./.inbox.index ./outbox ./.outbox.index ./sent-mail ./.sent-mail.index ./trash ./.trash.index ./drafts ./.drafts.index ... ./.lists.directory ... ./.lists.directory/zope ./.lists.directory/zope/new ./.lists.directory/zope/cur ./.lists.directory/zope/cur/1084688626.13214.23Wk:2,S ./.lists.directory/zope/cur/1084707019.13214.qrEX:2,S ... ./.lists.directory/zope/cur/1087378026.2080.H4Yb:2,S ./.lists.directory/zope/tmp ./.lists.directory/.zope.index ./.lists.directory/.zope.index.ids ... ./.lists.directory/debian-user ./.lists.directory/debian-user/new ./.lists.directory/debian-user/cur ./.lists.directory/debian-user/cur/1085365323.1786.Ui0D:2,S ./.lists.directory/debian-user/cur/1085365323.1786.Ejn3:2,S ... ./.djvm.index.sorted ./.srec.index.sorted ./.tmp.index.sorted --- Thanks again all, regards, James -- James Sinnamon [EMAIL PROTECTED] net au +61 412 319669, +61 2 95692123 (aka jaymz-.a.t.-bigpond-net-auStralia) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]