Q If it delivers when possible, won't the server flake if it get's 100,000 emails inject into Qmail? If I was to use your class and loop through 100,000 emails into Qmail, will Qmail attempt all at once?
A The answer was right above the question... "With qmail you do not have to run the queue like sendmail. You just inject the messages and it will deliver them when possible." It will build up a queue of everything it needs to send, then send them one at a time (or ten, or a hundred- you can configure that...) until it's done. It will never DoS itself by trying to send 10K eMails at once. Think of the queue as a long line. Mail waits its turn, then gets attention when it reaches the front of the line. Q BTW, isn't there any overhead associating with opening/closing a connection to inject each email into Qmail? Or it's very minimal overhead to even worry about? If 100,000 was looped and injected into Qmail, is Qmail fast enough that 100,000 emails are not just sitting on the server filling up disk space? A It's more than worth the overhead to use qmail. Basically, qmail does all the hard work for you. First of all, it sends multiple messages at once (default is 20, but it's configurable), so you're connection is never idle waiting for that SYN|ACK, DNS query, SMTP acknowledgement, etc. Now theoretically _could_ do this with PHP (pcntl), but why re-invent the MTA? It probably wouldn't be nearly as efficient, and it would take an eternity to develope a script that even approaches qmail's functionality. Now, with regards to a MLM (mailing list manager), you may want to take a look at ezmlm. You can find it (and qmail) at http://cr.yp.to/ It doesn't just delete addresses from the database automatically, but first sends a little probe eMail, threatening to delete the address if the probe bounces. It's a pretty cool feature, since it prevents good addresses from being dropped for stupid reasons. For instance, I get them sometimes (securityfocus runs ezmlm on bugtraq and vuln-dev, prolly their others too, and vulnwatch/vulndev both use it) when mail bounces due to MIME types that can be naughty (exe, vbs, com, eml, hehe .fon, .ttf, .otf soon?), or when someone uses profanity, or even if the DNS was temporarily down. Now personally i don't care about profanity or naughty MIME types, but lots of ISPs (like mine) filter them. Just something to think about. On Wednesday 22 January 2003 08:40 pm, Jonathan Chum wrote: > Hi, > > I was surfing through google's archive, and some reason my USENET reader > didn't catch this reply, so here goes. . .(my comments are embedded below) > > "Manuel Lemos" <[EMAIL PROTECTED]> wrote in message > [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... > > > Hello, > > > > On 12/22/2002 02:52 PM, Jonathan Chum wrote: > > >>>I was considering of writing the mass mailing application in PHP > > instead > > > >>>though. > > >>> > > >>>If anyone has eperience writing such applications with this amount of > > >>>emails, I'd like to know what you've done. > > >> > > >>If you do not need to send personalized messages (messages that differ > > >>for each recipient), just put all recipients in a BCc: header and send > > >> a single message to the local mailer queue (not via SMTP). > > >> > > >>If you do not care for the users that bounce messages, just make the > > >>return path be black hole email address. OTOH, if you care about > > >> bounces (you should if you mailing list is large or is not clean), > > >> consider using ezmlm, which is a mailing list manager than among other > > >> things takes care of bounce messages thanks to qmail VERP. I was told > > >> that is the one that eGroups hacked to use in the now known > > >> YahooGroups site. > > >> > > >>Once I built a small Web interface for ezmlm. It was meant just to > > >>create and edit several mailing lists meant to be used as newsletter > > >> for a portal with many sites. Is simple but it already comes with a > > >> SOAP interface to manage the mailing list subscribers remotely. > > >> > > >>http://www.phpclasses.org/ezmlmmanager > > > > > > I heard that using BCC, it can only handle a certain amount of > > receipients. > > > No, if you look in the archives of this list you will notice that this > > was explained several times. What happens is that some ISP of shared > > hosting limit the number of recipients of the messages you can send > > because they do not want you to do mass mailing at all. You should check > > your ISP Acceptable Use Policy before trying to do any mass mailing. > > Trying to queue individual messages to each recipient may fool the > > server limitations but you may still be against the AUP. > > > > > Also, messages that are BCC'd are also tend to be blocked by mail > > servers > > > > and clients. > > > > Any server owners that do that are dumb because while they think they > > are blocking spammers, what they get is they are simply blocking opt-in > > mailing lists for instance like this one you are getting. At the same > > time you still keep getting most of the spam because real spammers use > > dedicated machines make the recipients addresses show in the messages as > > needed to defeat that silly anti-spam criteria. So, what they get is to > > block solicited e-mail. > > > > Anyway, the problem of personalizing messages is that makes your > > mailings take much longer to generate and queue because you need to > > create separate message for each recipient. > > I'm weighing down personalization and enveloping for this project, and > probably going for the personalization of each email. This rules out BCC > and enveloping with SMTP. > > > >>No, queuing via SMTP is the slowest way to send messages. Your script > > >>should not bother to deliver the messages to the recipients SMTP > > >>servers. Delivery can take hours or days to finish due to network > > >>congestions and hard to conect SMTP servers. Just queue the messages in > > >>the local mailer and let it take care the actual delivery. > > >> > > >>I would recommend a qmail based system anytime, with or without ezmlm > > >> on top. In a production system that I manage, it just takes 3 seconds > > >> to queue a alert message to be sent to 50,000 via a local qmail > > >> server. > > >> > > >>You can also use sendmail almost as fast using the queue only mode. > > >> Some people think that sendmail is slow and many forks processes > > >> because they are not aware of how to configure it to queue the > > >> messages the fastest way that is possible. > > >> > > >>You may want to look into this class that has a sub-classes for > > >>delivering with sendmail program directly instead of using the mail(). > > >>It lets you configure the sendmail delivery mode. There is also a > > >>sub-class for delivering with qmail. > > >> > > >>http://www.phpclasses.org/mimemessage > > > > > > I was looking at a variety of methods, and seems that everyone thinks > > for > > > > best performance, you would inject the email in to Qmail or Sendmail's > > queue > > > That is because it is really the best. > > > :) Sounds good, definately going with Qmail on this project... > : > > > and run it every hour. > > > > With qmail you do not have to run the queue like sendmail. You just > > inject the messages and it will deliver them when possible. > > If it delivers when possible, won't the server flake if it get's 100,000 > emails inject into Qmail? If I was to use your class and loop through > 100,000 emails into Qmail, will Qmail attempt all at once? > > > > As for SMTP, I meant sending the email through your outbound SMTP > > > server which in our case, a server we build specifically for handling > > > that sort > > of > > > Of course all e-mail is delivered to each recipient via SMTP, but that > > you can relay to your local mailer. > > > > > volume. I was reading through Perl's BulkMail module, > > > http://mojo.skazat.com/support/documentation/Bulkmail.pm.html > > > > > > They are using SMTP to delivery their emails. Using envelopes and > > sorting > > > > the emails by their host meant even faster delivery. Though the > > > envelope > > > > That would be a good idea if you have messages to be sent to many > > recipients of the same domain but in reality it is more problematic. > > Some servers indeed reject messages to be sent many recipients at once. > > Although that is not illegal, it is a pattern of spamming. Another > > problem, is that you will have an hard time distinguishing which > > addresses of a same domain are bouncing and which are not bouncing. > > Very interesting... I hadn't thought about that. Definately don't want > legit emails mistaken as SPAM. > > > > method seems to be like BCC as you can't personalized each email To: > > > recepient. > > > > If you want delivery efficiency, forget personalization. > > > > > Your Mime Message class seems to be better for sending mass mailings as > > it > > > > queues up the email into Sendmail without opening/closing the > > > connection > > for > > > Actually the class provides different delivery methods that maybe > > appropriate in different circunstances: mail(), sendmail, qmail and SMTP. > > > > SMTP is the slowest for queuing but it is the fastest for direct > > delivery. My SMTP class supports direct deliveries which is great to > > deliver urgent messages as they are not relayed to the local mailer. It > > delivers to the recipient SMTP server and you will know if the message > > was accepted right away. I use this to deliver password reminder and > > other messages that users of my sites are anxious to receive because the > > local mailer queue may be loaded at the moment with some long delivery. > > That makes sense. > > > > each recepient. phpmailer.sourceforget.net has that problem of opening > > and > > > > closing a connection, yet they claim to receive good results of up to > > 40,000 > > > > emails per hour. Another software using that class was able to send > > 500,000 > > > > emails in 10 hours. > > > > You should not consider any delivery statistics because all depends on > > factors that have nothing to do with the software you use, like the > > outbound bandwidth, remote server reachability, anti-spam reception > > delays, temporary messages refusals (mailbox full, blocked account, > > etc..). > > Right. . . however I was more curious on a comparison between piping into > Qmail vs SMTP. So far, you have prooven greatly the differences and has > swayed my decision alot :) > > > > If injecting the emails directly into Qmail (Sounds to me that Qmail is > > far > > > > better for email deliver than to be using Sendmail) vs using SMTP, then > > > > DJB software rules! BTW, consider using DJBDNS cache to reduce DNS > > server lookup time. > > I'll check it out. We've been using Bind and are considering of using > TinyDNS, but also heard good things about DJBDNS. Mainly we are looking for > PostGreSQL extensions because we built a web hosting control panel that's > rounded around PostGresSQL to manage services. > > > > that'll be the approach. The only reason I'd would be using SMTP is to > > keep > > > > the code and data (on MySQL) stored on one machine and when it's time > > > to blast out an email, it'll establish a connection to a SMTP server > > assigned > > > > to the list to deliver the email. > > > > In that case, you may consider QMQP relay which is a protocol that lets > > you rely entire mail queues from a server to another. This is > > recommended when you have many busy mailing lists served from one > > machine and you can use more servers do the actual delivery. I think > > only qmail and ezmlm support QMQP. > > Thanks, I'll check QMQP as well. > > > > Though if the code was written to sit on each server that scans for > > pending > > > > tasks, it'll pick up it's job from a master database server and then > > > directly injecting Qmail locally. > > > > That is what qmail does. > > > > > The software app I'm writing is expected to have many users with small > > to > > > > very large lists. I'm trying to spec this out so that it scales well > > > and with the fastest delivery. From what I'm hearing from you and from > > > the > > first > > > > guy's reply to this thread, inject the email into Qmail is the quickest > > way. > > > Yes, qmail was thought for that. Yahoogroups use it and I have also used > > when I worked in a large portal with many subscriber newsletters as I > > mentioned. We had some very large like for MTV Brasil that had almost > > 400,000 subscribers. > > > > The greatest problem that made me learn some hard lessons is that it is > > very problematic if you start with subscribers list that are not clean > > up of no longer valid users. Once you start delivering messages to those > > addresses, you get a flood of bounces that pratically stop your server > > queue. > > > > The MTV newsletter was weekly, but that affected the newsletters of > > other sites that were stopped during the MTV newsletter bounces. Since > > ezmlm does not unsubscribe bouncing addresses right away, the solution > > was to remove bouncing subscribers using an external validation > > procedure. I used this other class for the cleanup. After that it was a > > breeze. We had peaks of 10,000 messages sent per minute. > > > > http://www.phpclasses.org/emailvalidation > > We had a 72K list with about 6K bad emails that Mailman didn't unsubscribe > until we used MojoMail which unsubscribes them from the database instantly > as well as perform validations. Now, blasts are handled alot quicker. > > So far, you have given me a good case to use Qmail and using your PHP class > as a foundation for this application. . . I'll have to check the other > software recommendations you made and see how'd I incorporate this into the > application. > > BTW, isn't there any overhead associating with opening/closing a connection > to inject each email into Qmail? Or it's very minimal overhead to even > worry about? If 100,000 was looped and injected into Qmail, is Qmail fast > enough that 100,000 emails are not just sitting on the server filling up > disk space? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
