Re: [Mailman-Users] Rewriting or identifying late bounces
* Brad Knowles b...@shub-internet.org wrote: on 7/8/09 6:12 PM, Stefan Förster said: Thanks for your advice, Brad. The problem is that, due to policy reasons, outgoing mail has to pass a content filter, running locally on the Mailman box. With VERP... Chuq von Rospach wrote some stuff in the FAQ detailing his experience with how VERP impacted performance on the systems he was managing. Of course, this doesn't necessarily apply directly to your case, but it is instructive to read. To be honest - I never understood where he got his numbers from. Or to be more precisely, how he got that lucky (note that I obfuscated the listname): # list_members mylist | cut -d@ -f2 | sort | uniq -c | sort -rn | head -3 1414 gmx.de 1009 web.de 404 freenet.de # list_members mylist | wc -l 2966 For that list, there would be 5 connections from Mailman to the local MTA for 2827 recipients, followed by = 139 connections for the remainder of the whole list, i.e. no more than 144 delivery attempts. With VERP, there would be 2966 connections from Mailman to the local MTA. With the content filter set up the way it is, without VERP, the MTA would create no more than 144*2 queue files and the content filter would in turn create 144 subdirectories with about 3 files each, for a total of 720 files and 144 directories created/unlink'd. VERP'd, I'd have to create/unlink 14830 files and 2966 directories. And yes, I know that the MTA might create more than one file per message delivery - I was assuming the best case. Perhaps I just got unlucky with the recipient distribution for that particular list - if it is of any importance to the lsit archives, I'd gladly investigate other lists I'm running. Or perhaps you could provide some live data from python.org lists? BTW - will MM3 be able to utilize the VERP features offered by many modern MTAs? I guess I will simply move the list server to another computer (and a different network). OTOH, moving the mailing list function to a different server and separating that from the content scanning system is also a good idea, including lots of other reasons. My main reason to move the whole installation is mainly that in another network, I don't have to scan every outgoing message. After all, it's not Mailman's fault that the box is heavily loaded. Good luck, and I hope that this works out for you. Thank you, Brad. Cheers Stefan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Rewriting or identifying late bounces
Stefan Förster wrote: BTW - will MM3 be able to utilize the VERP features offered by many modern MTAs? There are patches for Postfix which can probably be adapted for other MTAs. See the thread Patch for use of Postfix VERP support starting at http://mail.python.org/pipermail/mailman-users/2009-January/064588.html. This seems to be a good idea for future Mailman. I don't know if Barry has thought about it for MM3, but I will look at it for 2.2. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Rewriting or identifying late bounces
[Adding mailman-developers] On Jul 9, 2009, at 10:22 AM, Mark Sapiro wrote: Stefan Förster wrote: BTW - will MM3 be able to utilize the VERP features offered by many modern MTAs? There are patches for Postfix which can probably be adapted for other MTAs. See the thread Patch for use of Postfix VERP support starting at http://mail.python.org/pipermail/mailman-users/2009-January/064588.html . This seems to be a good idea for future Mailman. I don't know if Barry has thought about it for MM3, but I will look at it for 2.2. I have, and it doesn't look difficult to do, although I really want to rework the current delivery architecture so that it's more modular. This would also allow different lists to have different delivery mechanisms (e.g. bulk vs. individual) and should be more efficient and more extensible. VERPing is really only one half of the personalization or individual delivery story. Ideally it's the half that would by default always be enabled since it improves the reliability of your memberships. It's this part that would benefit greatly from adding MTA-specific support. I think Exim supports VERPing without an SMTP extension, and I get the impression it can be made to work in Sendmail too. The other half is really a mail merge operation, where the headers and footers (currently, but maybe someday the email body) contains placeholders that get filled in with personal information for the specific user. I'm not aware of a standard way to do this in any MTA, so I think if you want to do this kind of message body personalization, you still have to do that in Mailman. I'd like to support this better and would consider MTA-specific approaches. Mailman also supports something called full personalization which hacks other headers too. I put this in the category of mail merging since I'm not aware of any MTAs providing support for that either. -Barry PGP.sig Description: This is a digitally signed message part -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Rewriting or identifying late bounces
* Mark Sapiro m...@msapiro.net wrote: Stefan Förster wrote: generated by recipi...@intern.example.com mailbox is full: retry timeout exceeded Is there any way to rewrite those addresses or to help Mailman identify those bounces correctly? And yes, this a pretty stupid thing to do on their ISPs part... Mailman has a whole host of heuristic recognizers for non-compliant DSNs, but it doesn't try to guess if a non-member bouncing address might be a munged version of a member address. Because it's a pretty stupid thing to do - at least the fact that addresses are not rewritten twice. The 'best' solution for recognizing original recipients of bounced messages is Mailman's VERP like capability. This puts the original recipient's address in the envelope sender address so the envelope in this case is from invalidlist-bounces+recipient=example@lists.incertum.net. When the bounce is returned to that address, the MTA delivers it to the list-bounces address and Mailman recognizes the original recipient as recipi...@example.com from the envelope recipient of the DSN. The cost of this approach is that Mailman has to send an individual message to the outgoing MTA for each recipient. Bummer. There is no way to implement that, our mailing list server is already suffering from too much traffic, it's quite old hardware, after all. Thanks anyways Stefan -- Stefan Förster http://www.incertum.net/ Public Key: 0xBBE2A9E9 FdI #283: whoami - Whoami ist nur was für Leute mit Alzheimer! (Begründung von Microsoft Deutschland für das Entfernen des Befehls aus Windows NT) -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Rewriting or identifying late bounces
on 7/8/09 11:56 AM, Stefan Förster said: Bummer. There is no way to implement that, our mailing list server is already suffering from too much traffic, it's quite old hardware, after all. Take a look at the stuff on performance tuning in the FAQ. Even really old hardware can perform amazingly well, if it's tuned correctly. I guarantee that you would be astonished at what I can achieve with an ancient Compaq Pentium-133 laptop with 64MB of RAM and a 10GB hard drive (upgraded from 1GB). If you've already done as much performance tuning as you can do, then obviously you're not going to squeeze any more blood out of that turnip. But I would suggest that you familiarize yourself with these sections to make sure that you're actually doing as much as you can. -- Brad Knowles b...@shub-internet.org LinkedIn Profile: http://tinyurl.com/y8kpxu -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Rewriting or identifying late bounces
* Brad Knowles b...@shub-internet.org wrote: on 7/8/09 11:56 AM, Stefan Förster said: Bummer. There is no way to implement that, our mailing list server is already suffering from too much traffic, it's quite old hardware, after all. Take a look at the stuff on performance tuning in the FAQ. Even really old hardware can perform amazingly well, if it's tuned correctly. I guarantee that you would be astonished at what I can achieve with an ancient Compaq Pentium-133 laptop with 64MB of RAM and a 10GB hard drive (upgraded from 1GB). If you've already done as much performance tuning as you can do, then obviously you're not going to squeeze any more blood out of that turnip. But I would suggest that you familiarize yourself with these sections to make sure that you're actually doing as much as you can. Thanks for your advice, Brad. The problem is that, due to policy reasons, outgoing mail has to pass a content filter, running locally on the Mailman box. With VERP... I guess I will simply move the list server to another computer (and a different network). Cheers Stefan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Rewriting or identifying late bounces
on 7/8/09 6:12 PM, Stefan Förster said: Thanks for your advice, Brad. The problem is that, due to policy reasons, outgoing mail has to pass a content filter, running locally on the Mailman box. With VERP... Chuq von Rospach wrote some stuff in the FAQ detailing his experience with how VERP impacted performance on the systems he was managing. Of course, this doesn't necessarily apply directly to your case, but it is instructive to read. My recollection is that, in his case, he found that without VERP he got about two recipients per copy of each message transmitted -- due to the fact that some recipients are all on the same system and only one copy is sent to that system for multiple recipients, while others get unique copies because no one else is subscribed from that system. That meant that enabling VERP roughly doubled the number of copies of messages that had to be sent (so that each person is guaranteed to get their own personal unique copy), but that this didn't actually affect the overall performance very much (since so much of e-mail is I/O bound and waiting for the system at the other end to respond). However, enabling VERP also meant that it was now much, much easier for the system to automatically manage bounces (a.k.a., Non-Delivery Notices, or NDNs), delivery status notices (DSNs), etc This made overall management of the system much easier, and greatly reduced the amount of work that the system had to try to do to parse the bounces to try to figure out which recipient(s) it was in relation to. If you throw a content scanning system into that mix, most of the content of each of those mailing list messages will be the same, so depending on how that content scanning system is configured, it shouldn't be that much more expensive to process 100 virtually identical messages as it is to process the first message in that group. I guess I will simply move the list server to another computer (and a different network). OTOH, moving the mailing list function to a different server and separating that from the content scanning system is also a good idea, including lots of other reasons. Good luck, and I hope that this works out for you. -- Brad Knowles b...@shub-internet.org LinkedIn Profile: http://tinyurl.com/y8kpxu -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Users] Rewriting or identifying late bounces
Hello world, some of my list members have a mail address like recipi...@example.com - nothing special about that. When mail is delivered to their address, their ISPs mail server rewrite the address internally to recipi...@intern.example.com and try to deliver them to the users mailbox. If for any reason, e.g. quota, this fails, a NDR/DSN is sent with the new name: Received: from mout3.example.com (mout3.example.com [195.4.92.93]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.incertum.net (Postfix) with ESMTPS for invalidlist-boun...@lists.incertum.net; Tue, 7 Jul 2009 23:09:54 +0200 (CEST) Received: from [195.4.92.10] (helo=0.mx.example.com) by mout3.example.com with esmtpa (ID exim) (port 25) (Exim 4.69 #92) id 1MOHvF-000306-UO for invalidlist-boun...@lists.incertum.net; Tue, 07 Jul 2009 23:09:53 +0200 Received: from mbox134.example.com ([195.4.93.134]:43037) by 0.mx.example.com with esmtpa (ID exim) (port 25) (Exim 4.69 #79) id 1MOHvF-0001Qz-H2 for invalidlist-boun...@lists.incertum.net; Tue, 07 Jul 2009 23:09:53 +0200 Received: from exim by mbox134.example.com with local (Exim 4.69 #91) id 1MOHvF-0007qp-Dc for invalidlist-boun...@lists.incertum.net; Tue, 07 Jul 2009 23:09:53 +0200 X-Failed-Recipients: recipi...@intern.example.com Auto-Submitted: auto-replied From: Mail Delivery System mailer-dae...@example.com To: invalidlist-boun...@lists.incertum.net Subject: Mail delivery failed: returning message to sender Message-Id: e1mohvf-0007qp...@mbox134.example.com Date: Tue, 07 Jul 2009 23:09:53 +0200 X-purgate-ID: 149285::1247000993-22A6-9D72E1E7/0-0/0-18 This message was created automatically by mail delivery software. A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed: save to inbox generated by recipi...@intern.example.com mailbox is full: retry timeout exceeded Is there any way to rewrite those addresses or to help Mailman identify those bounces correctly? And yes, this a pretty stupid thing to do on their ISPs part... Ciao Stefan -- Stefan Förster http://www.incertum.net/ Public Key: 0xBBE2A9E9 You're here because you know something...you can't explain but you feel it... that there's something wrong with the world. --Morpheus, from The Matrix -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Users] Rewriting or identifying late bounces
Stefan Förster wrote: some of my list members have a mail address like recipi...@example.com - nothing special about that. When mail is delivered to their address, their ISPs mail server rewrite the address internally to recipi...@intern.example.com and try to deliver them to the users mailbox. If for any reason, e.g. quota, this fails, a NDR/DSN is sent with the new name: [...] X-Failed-Recipients: recipi...@intern.example.com Auto-Submitted: auto-replied From: Mail Delivery System mailer-dae...@example.com To: invalidlist-boun...@lists.incertum.net Subject: Mail delivery failed: returning message to sender Message-Id: e1mohvf-0007qp...@mbox134.example.com Date: Tue, 07 Jul 2009 23:09:53 +0200 X-purgate-ID: 149285::1247000993-22A6-9D72E1E7/0-0/0-18 This message was created automatically by mail delivery software. A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed: save to inbox generated by recipi...@intern.example.com mailbox is full: retry timeout exceeded Is there any way to rewrite those addresses or to help Mailman identify those bounces correctly? And yes, this a pretty stupid thing to do on their ISPs part... Mailman has a whole host of heuristic recognizers for non-compliant DSNs, but it doesn't try to guess if a non-member bouncing address might be a munged version of a member address. The 'best' solution for recognizing original recipients of bounced messages is Mailman's VERP like capability. This puts the original recipient's address in the envelope sender address so the envelope in this case is from invalidlist-bounces+recipient=example@lists.incertum.net. When the bounce is returned to that address, the MTA delivers it to the list-bounces address and Mailman recognizes the original recipient as recipi...@example.com from the envelope recipient of the DSN. The cost of this approach is that Mailman has to send an individual message to the outgoing MTA for each recipient. -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
