Re: [Mailman-Users] High usage -- making Mailman nice
On Thu, 2005-12-08 at 17:50 -0800, Mark Sapiro wrote: John Dennis wrote: If your MTA can only handle one, or a small number of client connections then each connection will be busy handling SMTP_MAX_RCPTS and other client connections will queue up. If you have 4,000 recipients then you have the potential to tie up 8 SMTP client connections. I don't think this is correct. Unless there are multiple outgoing runners processing slices, I don't see how SMTP delivery is multi-threaded, and even if there are multiple runners, a single message to even 4000 recipients is going to be processed in its entirety by one runner. Depends on whether the client waits for a success status or not, which if my memory serves me correctly (its been failing lately :-) is not the behavior in an SMTP transaction. I believe the MTA accepts the input, queues the request and control is returned, at that point I believe the connection is typically closed. At this point the outgoing running will loop again and attempt initiate a new SMTP transaction. At least that how I think it works, plus the behavior is specific to each MTA. If the client waits for the transaction to complete they of course you're right. Mailman is not multi-threaded but many MTA's are or more properly many MTA's pre-fork a pool of processes which are managed like threads. -- John Dennis [EMAIL PROTECTED] -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=showamp;file=faq01.027.htp
Re: [Mailman-Users] High usage -- making Mailman nice
John Dennis wrote: Depends on whether the client waits for a success status or not, which if my memory serves me correctly (its been failing lately :-) is not the behavior in an SMTP transaction. I believe the MTA accepts the input, queues the request and control is returned, at that point I believe the connection is typically closed. At this point the outgoing running will loop again and attempt initiate a new SMTP transaction. At least that how I think it works, plus the behavior is specific to each MTA. Mailman, via SMTPDirect sends all the delivery chunks for a message before returning control to the outgoing runner. Assuming we're not VERPing or personalizing, it sends the 'MAIL FROM' command followed by SMTP_MAX_RCPTS 'RCPT TO' commands, followed by 'DATA' and the message. After receiving the response to the 'DATA', it (normally, but depends on SMTP_MAX_SESSIONS_PER_CONNECTION) sends the 'MAIL FROM' for the next chunk on the same connection. It may throw in another HELO/EHLO between transactions. This is in the Python smtplib, and I haven't looked in detail. If the client waits for the transaction to complete they of course you're right. The client (Mailman) waits for the response to the 'DATA' command for each chunk. This doesn't mean the mail has been sent, but it has been accepted and queued within the MTA. How many queues the MTA uses at this point to actually send the mail is up to the MTA, not Mailman. -- Mark Sapiro [EMAIL PROTECTED] The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=showamp;file=faq01.027.htp
[Mailman-Users] High usage -- making Mailman nice
Morning, We're running MM v2.1.4-4, we're experiencing such high usage on our dual xeon server when mailman sends to 4000 users that port 25 becomes unresponsive. is there a way to nice the mailman delivery? Caylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=showamp;file=faq01.027.htp
Re: [Mailman-Users] High usage -- making Mailman nice
On Thu, 2005-12-08 at 11:10 -0600, Caylan Van Larson wrote: Morning, We're running MM v2.1.4-4, we're experiencing such high usage on our dual xeon server when mailman sends to 4000 users that port 25 becomes unresponsive. is there a way to nice the mailman delivery? It sounds like your MTA is inefficient in some manner, is not a huge load for a dual server backed with a high bandwith connection. How many client connections can it simultaneously handle? Any chance you've got DNS delays (especially reverse lookup)? There are ways to control how many recipients are sent in each SMTP transaction. SMTP_MAX_RCPT = 500 is the default in mailman, defined in Defaults.py, overridden in mm_cfg.py If your MTA can only handle one, or a small number of client connections then each connection will be busy handling SMTP_MAX_RCPTS and other client connections will queue up. If you have 4,000 recipients then you have the potential to tie up 8 SMTP client connections. If you raise SMTP_MAX_RCPTS you may exceed your MTA's limit on number of recipients, but you'll consume few connections increasing the pool of available connections. If you lower it you'll increase the total connections needed, but your MTA should take connections round robin allowing other to jump in. By all means verify how many client connections your SMTP can handle. If you have VERP or personalization turned on mailman will have to initiate one SMTP transaction per recipient (are these features turned on?). This is inefficient because there is one recipient per SMTP connection. It will take the SMTP much longer to process because of the higher overhead, but on the other hand there should be plenty of opportunity for other clients to obtain a connection from the SMTP connection pool because mailman will cycle through connections in a serial manner leaving the rest of the SMTP client connection pool available for other SMTP clients. HTH, -- John Dennis [EMAIL PROTECTED] -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=showamp;file=faq01.027.htp
Re: [Mailman-Users] High usage -- making Mailman nice
At 1:12 PM -0500 2005-12-08, John Dennis wrote: We're running MM v2.1.4-4, we're experiencing such high usage on our dual xeon server when mailman sends to 4000 users that port 25 becomes unresponsive. is there a way to nice the mailman delivery? It sounds like your MTA is inefficient in some manner, is not a huge load for a dual server backed with a high bandwith connection. How many client connections can it simultaneously handle? Any chance you've got DNS delays (especially reverse lookup)? They could also be doing all sorts of authentication/authorization or anti-spam/anti-virus checks on their outgoing mail. It is clearly stated in the FAQ that this is generally a bad idea for outgoing mail from the mailing list, since all that stuff should have been checked on the way in. The OP would be well-served to go to the Mailman FAQ Wizard at http://www.python.org/cgi-bin/faqw-mm.py and search for performance, and read all the related FAQ entries. If you go down that list, I bet you'll find the problem and be able to relatively easily fix it. -- Brad Knowles, [EMAIL PROTECTED] Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. -- Benjamin Franklin (1706-1790), reply of the Pennsylvania Assembly to the Governor, November 11, 1755 LOPSA member since December 2005. See http://www.lopsa.org/. -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=showamp;file=faq01.027.htp
Re: [Mailman-Users] High usage -- making Mailman nice
On Dec 8, 2005, at 12:49 PM, Brad Knowles wrote: At 1:12 PM -0500 2005-12-08, John Dennis wrote: We're running MM v2.1.4-4, we're experiencing such high usage on our dual xeon server when mailman sends to 4000 users that port 25 becomes unresponsive. is there a way to nice the mailman delivery? It sounds like your MTA is inefficient in some manner, is not a huge load for a dual server backed with a high bandwith connection. How many client connections can it simultaneously handle? Any chance you've got DNS delays (especially reverse lookup)? They could also be doing all sorts of authentication/authorization or anti-spam/anti-virus checks on their outgoing mail. It is clearly stated in the FAQ that this is generally a bad idea for outgoing mail from the mailing list, since all that stuff should have been checked on the way in. Thanks for all those that responded. After thinking about this for a bit, we realized that we were using a content-filter in postfix to divert all incoming port 25 to a barracuda spam appliance. This was then coming back in 10027 for delivery. To further amplify the problem, /home is NFS mounted... which means, lots of network IO. Here are the changes we made. mm_cfg.py SMTP_MAX_RCPT = 2000 SMTPPORT=10027 main.cf smtpd_recipient_limit = 1000 default_process_limit = 150 Thanks, Caylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=showamp;file=faq01.027.htp
Re: [Mailman-Users] High usage -- making Mailman nice
Caylan Van Larson wrote: Here are the changes we made. mm_cfg.py SMTP_MAX_RCPT = 2000 SMTPPORT=10027 main.cf smtpd_recipient_limit = 1000 default_process_limit = 150 I'm not certain about this, but this looks wrong to me. You're telling Mailman to send to 2000 recipients in one transaction, and it appears you're telling the MTA to only accept up to 1000. -- Mark Sapiro [EMAIL PROTECTED] The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=showamp;file=faq01.027.htp
Re: [Mailman-Users] High usage -- making Mailman nice
John Dennis wrote: If your MTA can only handle one, or a small number of client connections then each connection will be busy handling SMTP_MAX_RCPTS and other client connections will queue up. If you have 4,000 recipients then you have the potential to tie up 8 SMTP client connections. I don't think this is correct. Unless there are multiple outgoing runners processing slices, I don't see how SMTP delivery is multi-threaded, and even if there are multiple runners, a single message to even 4000 recipients is going to be processed in its entirety by one runner. -- Mark Sapiro [EMAIL PROTECTED] The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=showamp;file=faq01.027.htp
Re: [Mailman-Users] High usage -- making Mailman nice
On Dec 8, 2005, at 7:43 PM, Mark Sapiro wrote: Caylan Van Larson wrote: Here are the changes we made. mm_cfg.py SMTP_MAX_RCPT = 2000 SMTPPORT=10027 main.cf smtpd_recipient_limit = 1000 default_process_limit = 150 I'm not certain about this, but this looks wrong to me. You're telling Mailman to send to 2000 recipients in one transaction, and it appears you're telling the MTA to only accept up to 1000. You're right, there was a typo in the email. The configs for both say 2000. :-) Caylan -- Mailman-Users mailing list Mailman-Users@python.org http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=showamp;file=faq01.027.htp
