RE: Mailing from One connection
I am sending different mails to 20,000 recipients at a time. So, each qmail-remote sends a mail to each recipient. or am I wrong No, that's more-or-less how it works. If in the total 20,000 mails, say 5000 are hotmail, 5000 are yahoo and the rest are to other domains. Then, is it possible to open a single qmail-remote process and dump all messages to be sent to hotmail on one connection and open another connection for all yahoo messages Not without alot of patching. I guess this speeds up the mail delivery amazingly Nope. It's pretty much a myth. Maybe you gain a little by opening fewer connections, but chances are some of them will timeout in the process and have to start over. This is covered in the archives, and discussed at some length in the ezmlm mailing list archives. --joshua.
Re: Mailing from One connection
D Rajesh [EMAIL PROTECTED] wrote: If in the total 20,000 mails, say 5000 are hotmail, 5000 are yahoo and the rest are to other domains. Then, is it possible to open a single qmail-remote process and dump all messages to be sent to hotmail on one connection and open another connection for all yahoo messages qmail is designed specifically _not_ to do this. sendmail does this. I guess this speeds up the mail delivery amazingly No, it slows it down tremendously. That's why qmail doesn't do it. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ ---
Re: Mailing from One connection
you're asking for the perfect spam solution your targets get mail messages without a To: header or a bogus one, and if they're lucky enough to receive mail on one address but reply with another, you'll never know who's it coming from it's clearly not the rightway to do things dan - Original Message - From: D Rajesh To: [EMAIL PROTECTED] Sent: Monday, July 09, 2001 2:28 PM Subject: Mailing from One connection Hi All, I am sending different mails to 20,000recipients at a time. So, each qmail-remote sends a mail to each recipient. or am I wrong If in the total 20,000 mails, say 5000 are hotmail, 5000 are yahoo and the rest are to other domains. Then, is it possible to open a single qmail-remote process and dump all messages to be sent to hotmail on one connection and open another connection for all yahoo messages I guess this speeds up the mail delivery amazingly Cheers, rajesh.
Re: Mailing from One connection
Charles Cazabon [EMAIL PROTECTED] D Rajesh [EMAIL PROTECTED] wrote: DR If in the total 20,000 mails, say 5000 are hotmail, 5000 are yahoo and the DR rest are to other domains. Then, is it possible to open a single DR qmail-remote process and dump all messages to be sent to hotmail on one DR connection and open another connection for all yahoo messages CC qmail is designed specifically _not_ to do this. sendmail does this. DR I guess this speeds up the mail delivery amazingly CC No, it slows it down tremendously. That's why qmail doesn't do it. * From Rodney: I haven't tested qmail against sendmail, infact I've barely even used qmail. But I have done a bit of network programming, so let me open a few points for consideration: === If you send a single messager to a given domain, then the activity on your side should look something like this: - Lookup recipient domain. This requires getting the current MX for the recipient. - Open a socket to the mail server for the recipient domain (connect). - Tell the recipient mail server who you are. (trivial, but existant) - Send the mail (headers and body) - Close the socket connection. According to the input from Charles, qmail repeats this entire process for EVERY message sent to a given domain. === If you handle a bunch of mail to this same domain with a single connection (we'll assume for the moment that the body is the same for each message), then the activity on your side should look something like this: - Lookup recipient domain. This requires getting the current MX for the recipient. - Open a socket to the mail server for the recipient domain (connect). - Tell the recipient mail server who you are. (trivial, but existant) - Send the mail (headers, a potentially really long recipient list, and body) - Close the socket connection. Hmm, only one copy of the body got sent to 5000 recipients. Let's see, 5000 times 1K... Hey, that's 5 megabytes! === OK, to be fair I'll include the example of sending a bunch of mail to this same domain with a single connection but with differing message bodies. - Lookup recipient domain. This requires getting the current MX for the recipient. - Open a socket to the mail server for the recipient domain (connect). - Tell the recipient mail server who you are. (trivial, but existant) - For each message: * Send the mail (headers and body) - Close the socket connection. See, we've missed 5000 name lookups and socket connections. I'll admit that a decent name server helps this. Also, that creating a socket isn't very expensive on today's hardware, but it's at least a little bit of the load. What will become expensive is establishing the actuall connection with the remote machine (which happens after the socket is created and before data is sent). Something else to consider is manually sending all of the messages from your program. Not too tough, in Perl you'd just use Net::SMTP, do the afore mentioned name lookup (I use Net::DNS for this), get a connection to that server, send your messages, then close that connection. The reason for doing it this way is mostly to save on the total number of processes spawned on your machine. The importance of this will depend heavily on the power and load level of your system. === So, after all of that, here's my suggestion to you Mr/Ms Rajesh (dunno which, you didn't give your first name): If you haven't already done so, install sendmail. You don't need to configure it to receive, which means that it won't conflict with your running qmail. Then, use sendmail for your MUA. Make sure that you have sendmail's queue set up to a long enough time to let your program transfer all of the messages into sendmail. I would guess that 10 minutes would be plenty for 2 messages at a stroke. That way when sendmail wakes up and goes to sending mail, he'll handle tasks like grouping messages and sending with fewer connections. If you acutally do these tests, please let us know what happens. I'm very interested in hearing input on performance from an impartial person. That is; not partial to qmail or sendmail. (I'm slowly putting aside my sendmail bias. ;-) ) --- Rodney Broom Programmer: Desert.Net
Re: Mailing from One connection
Rodney Broom [EMAIL PROTECTED] writes: === OK, to be fair I'll include the example of sending a bunch of mail to this same domain with a single connection but with differing message bodies. Note that VERP, which is very useful for mailing lists, requires this approach, because the one connection approach doesn't permit a different envelope sender for each recipient. - Lookup recipient domain. This requires getting the current MX for the recipient. - Open a socket to the mail server for the recipient domain (connect). - Tell the recipient mail server who you are. (trivial, but existant) - For each message: * Send the mail (headers and body) - Close the socket connection. See, we've missed 5000 name lookups and socket connections. I'll admit that a decent name server helps this. Also, that creating a socket isn't very expensive on today's hardware, but it's at least a little bit of the load. What will become expensive is establishing the actuall connection with the remote machine (which happens after the socket is created and before data is sent). You have correctly demonstrated that an approach other than the qmail uses will reduce the number of bytes sent over the network. If you pay for your network usage by the byte, then this could be important. For people with a flat rate connection, however, minimizing message delivery latency is usually more interesting. qmail sends many messages in parallel, which helps to overcome the round trip latency inherent in the SMTP protocol. Overall, the messages will be delivered faster using qmail. This has all been discussed in the archives before. I encourage you to check them. Ian
Re: Mailing from One connection
Rodney Broom [EMAIL PROTECTED] wrote: Charles Cazabon [EMAIL PROTECTED] D Rajesh [EMAIL PROTECTED] wrote: re: batched recipients per-MX or per-domain DR I guess this speeds up the mail delivery amazingly CC No, it slows it down tremendously. That's why qmail doesn't do it. === If you send a single messager to a given domain, then the activity on your side should look something like this: - Lookup recipient domain. This requires getting the current MX for the recipient. No network load after the first one; it will be in your DNS cache. You are running a local DNS cache on every mail server, aren't you? - Open a socket to the mail server for the recipient domain (connect). On a good OS, this is negligible, and qmail is very efficient. - Tell the recipient mail server who you are. (trivial, but existant) - Send the mail (headers and body) - Close the socket connection. None of this is different with per-domain batching. According to the input from Charles, qmail repeats this entire process for EVERY message sent to a given domain. Yes. It's faster because it parallelizes well, and you have fewer round-trip waits. With batching, you have to do: send one RCPT wait for response send next RCPT wait for response ... By the time you've listed ten or twenty recipients, qmail has finished delivering all ten or twenty copies in parallel. Hmm, only one copy of the body got sent to 5000 recipients. Let's see, 5000 times 1K... Hey, that's 5 megabytes! qmail is designed for well-connected hosts. If you want to send a message to 20k recipients over a narrowband line, don't use qmail. See, we've missed 5000 name lookups and socket connections. I'll admit that a decent name server helps this. Also, that creating a socket isn't very expensive on today's hardware, but it's at least a little bit of the load. Not on my systems -- qmail simply is a negligible load on a decent OS, even when sitting at a constant concurrencyremote of 100-150. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ ---
Re: Mailing from One connection
Rodney Broom [EMAIL PROTECTED] writes: Charles Cazabon [EMAIL PROTECTED] D Rajesh [EMAIL PROTECTED] wrote: DR If in the total 20,000 mails, say 5000 are hotmail, 5000 are yahoo and the DR rest are to other domains. Then, is it possible to open a single DR qmail-remote process and dump all messages to be sent to hotmail on one DR connection and open another connection for all yahoo messages CC qmail is designed specifically _not_ to do this. sendmail does this. DR I guess this speeds up the mail delivery amazingly CC No, it slows it down tremendously. That's why qmail doesn't do it. * From Rodney: I haven't tested qmail against sendmail, infact I've barely even used qmail. But I have done a bit of network programming, so let me open a few points for consideration: [...] This has been hashed, rehashed, and re-re-hashed on this list. It inevitably ends in a flameware, somebody telling somebody else to profile rather then speculate, and a series of past analyses of these events supporting both sides of the argument being dredged up. Please don't continue with this unless you have read through the archives and know what you're getting into. :) -ScottG.
Re: Mailing from One connection
Rodney Broom [EMAIL PROTECTED] writes: This has been hashed, rehashed, and re-re-hashed on this list. It inevitably ends in a flameware, somebody telling somebody else to profile rather then speculate, and a series of past analyses of these events supporting both sides of the argument being dredged up. Test with stock qmail on a Solaris workstation, 10,000 copies sent to the same email address (obviously the same domain) using qmail-inject: 30 minutes. Test from same workstation with a script to generate 10,000 rcpt to: lines and send via a single connection: 5 minutes. In the first example, 10,000 actual copies were delivered to the mailbox but in the second, only a single copy was delivered. Presuming it should take the same amount of time to wait for a rcpt to: response whether sending a separate message at a time or a single message with multiple rcpt to: lines, I get the results that I expected - to send to the same domain (ignoring VERP requirements), it is faster to use a single connection for multiple messages than to use qmail. -- Roger Walker Tier III Messaging/News Team Internet Applications, National Consumer IP TELUS Corporation 780-493-2471
Re: Mailing from One connection
Dave Sill [EMAIL PROTECTED] wrote: -Dave That's bizarre. What I actually sent was: http://www.lifewithqmail.org/lwq.html#multi-rcpt -Dave -Dave
Re: Mailing from One connection
Roger Walker [EMAIL PROTECTED] wrote: [snip bogus test] I get the results that I expected - to send to the same domain (ignoring VERP requirements), it is faster to use a single connection for multiple messages than to use qmail. Fine. Don't use qmail. This discussion is closed. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ ---
Re: Mailing from One connection
I beg to differ... I have a list of 40k I'll use to race ya.. Hell I'll even let you use a list of 10k to race my list of 40k.. Me using qmail and you using Sendmail.. I'll even go beyond that I'll limit the bandwidth my server can consume to 1600kb/s and you can use whatever you want.. I'll still win.. Speed is not the key here. The key is in how mail is handled. Sendmail opens up a single connection and dumps all similiar address for that domain into that connection one obvious slow down is in response time waiting for the server to acknoledge the User address, accept the message and then reset. Whereas qmail just cranks out each message in it's own instance and does not have to deal with all those extra commands and can open as many as (in my case) 400 connections to a single remote server at one shoot, limiting my bandwidth to 1600k I can still crank out something like 1400 messages a minute sustained using qmail.. Can Sendmail do the same?? I think not. --JT - Original Message - From: Roger Walker [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, July 09, 2001 11:30 AM Subject: Re: Mailing from One connection Rodney Broom [EMAIL PROTECTED] writes: This has been hashed, rehashed, and re-re-hashed on this list. It inevitably ends in a flameware, somebody telling somebody else to profile rather then speculate, and a series of past analyses of these events supporting both sides of the argument being dredged up. Test with stock qmail on a Solaris workstation, 10,000 copies sent to the same email address (obviously the same domain) using qmail-inject: 30 minutes. Test from same workstation with a script to generate 10,000 rcpt to: lines and send via a single connection: 5 minutes. In the first example, 10,000 actual copies were delivered to the mailbox but in the second, only a single copy was delivered. Presuming it should take the same amount of time to wait for a rcpt to: response whether sending a separate message at a time or a single message with multiple rcpt to: lines, I get the results that I expected - to send to the same domain (ignoring VERP requirements), it is faster to use a single connection for multiple messages than to use qmail. -- Roger Walker Tier III Messaging/News Team Internet Applications, National Consumer IP TELUS Corporation 780-493-2471
Re: Mailing from One connection
I beg to differ... Only quoted the evidence... Wheras qmail just cranks out each message in it's own instance and does not have to deal with all those extra commands and can open as many as (in my case) 400 connections to a single remote server at one shoot, limiting my bandwidth to 1600k I can still crank out something like 1400 messages a minute sustained using qmail.. Can Sendmail do the same?? I think not. I didn't say anything about sendmail. All I did was: telnet smtp.host 25 mail.test mail.test contained the smtp conversation and message body, including all 10,000 rcpt to: lines. This had nothing to do with how fast qmail or anything else could send, I think, but more with how fast the remote system received, whether by my script or by qmail (either from the same workstation). -- Roger Walker Tier III Messaging/News Team Internet Applications, National Consumer IP TELUS Corporation 780-493-2471
Re: Mailing from One connection
* Dave Sill [EMAIL PROTECTED] [010709]: Dave Sill [EMAIL PROTECTED] wrote: -Dave That's bizarre. What I actually sent was: http://www.lifewithqmail.org/lwq.html#multi-rcpt -Dave -Dave If I could make a guess, in the original message you had http: www.lifewithqmail.org/lwq.html#multi-rcpt right under the headers.I would imagine it was seen as a header and possibly hidden from view. -Ryan
Re: Mailing from One connection
On Mon, Jul 09, 2001 at 12:30:52PM -0600, Roger Walker wrote: Test with stock qmail on a Solaris workstation, 10,000 copies sent to the same email address (obviously the same domain) using qmail-inject: 30 minutes. Test from same workstation with a script to generate 10,000 rcpt to: lines and send via a single connection: 5 minutes. In the first example, 10,000 actual copies were delivered to the mailbox but in the second, only a single copy was delivered. Presuming it should take the same amount of time to wait for a rcpt to: response whether sending a separate message at a time or a single message with multiple rcpt to: lines, I get the results that I expected - to send to the same domain (ignoring VERP requirements), it is faster to use a single connection for multiple messages than to use qmail. Amazing! I guess you're right. What is this MTA called? Where can I download it? Let me know, and I'll set it up on a test box to try to duplicate your test. What were the IP addresses of the two boxes you did this on? What kind of dns library does this server use for it's resolution? And what was the name again? What server did it use for the resolution, and what was the dns latency for that from the sending boxes? I'm going to try to duplicate your test as closely as possible. What was the ip of that dns server again? Wait, I'm reading your post a bit more closely and it doesn't look like you benchmarked qmail against your server, but against a script to generate 10K rcpt to: lines. Is that right? Now I'm a bit confused. qmail is an MTA which handles many things like a safe queue. What are you comparing that to? The case where I have 10K recipients of one message at one domain which never needs queue management? How does your script handle new messages? How does your script handle a randomly mixed list of 10K recipients who are located at 10 different domains? How does your script handle a list of 50M recipients at one domain? Does your script accept message via the smtp protocol? If so, what happens after it replys ok to the 50M message case, and you power off the box 5 seconds later? Can you send me the source of this script? John
Re: Mailing from One connection
Hi All, Thank you to all for all the replies I also found that Qmail is definitely better than sendmail. After reading lot of documentation links given here and elsewhere I also found that sending mails concurrently is better than sending mails from one connection. I have checked the mailing lists and archives on the same too. Cheers Rajesh. - Original Message - From: John White [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Tuesday, July 10, 2001 9:44 AM Subject: Re: Mailing from One connection On Mon, Jul 09, 2001 at 12:30:52PM -0600, Roger Walker wrote: Test with stock qmail on a Solaris workstation, 10,000 copies sent to the same email address (obviously the same domain) using qmail-inject: 30 minutes. Test from same workstation with a script to generate 10,000 rcpt to: lines and send via a single connection: 5 minutes. In the first example, 10,000 actual copies were delivered to the mailbox but in the second, only a single copy was delivered. Presuming it should take the same amount of time to wait for a rcpt to: response whether sending a separate message at a time or a single message with multiple rcpt to: lines, I get the results that I expected - to send to the same domain (ignoring VERP requirements), it is faster to use a single connection for multiple messages than to use qmail. Amazing! I guess you're right. What is this MTA called? Where can I download it? Let me know, and I'll set it up on a test box to try to duplicate your test. What were the IP addresses of the two boxes you did this on? What kind of dns library does this server use for it's resolution? And what was the name again? What server did it use for the resolution, and what was the dns latency for that from the sending boxes? I'm going to try to duplicate your test as closely as possible. What was the ip of that dns server again? Wait, I'm reading your post a bit more closely and it doesn't look like you benchmarked qmail against your server, but against a script to generate 10K rcpt to: lines. Is that right? Now I'm a bit confused. qmail is an MTA which handles many things like a safe queue. What are you comparing that to? The case where I have 10K recipients of one message at one domain which never needs queue management? How does your script handle new messages? How does your script handle a randomly mixed list of 10K recipients who are located at 10 different domains? How does your script handle a list of 50M recipients at one domain? Does your script accept message via the smtp protocol? If so, what happens after it replys ok to the 50M message case, and you power off the box 5 seconds later? Can you send me the source of this script? John
