Re: open(FH,'|qmail-inject') fails
Greg Stark wrote: > A better plan for such systems is to have a queue in your database for > parameters for e-mails to send. Insert a record in the database and let your > web server continue processing. > > Have a separate process possibly on a separate machine or possibly on multiple > machines do selects from that queue and deliver mail. I think the fastest way > is over a single SMTP connection to the mail relay rather than forking a > process to inject the mail. > > This keeps the very variable -- even on your own systems -- mail latency > completely out of the critical path for web server requests. Which is really > the key measure that dictates the requests/s you can serve. > Exactly, Greg. This is homologous to proxy serving http requests. Ideally, the data/text should be relayed to a separate, dedicated mail server. This has come up repeatedly for me on performance tuning projects. If there are a number of mail processes negotiating with remote hosts even running on the same machine as you are web serving from, you may, under significant load, degrade performance.
Re: open(FH,'|qmail-inject') fails
Doug MacEachern <[EMAIL PROTECTED]> writes: > On Mon, 25 Sep 2000, Stas Bekman wrote: > > > All you care about is to measure the time between email sending start and > > end (when the process continues on its execution flow). Why should one > > care about the details of internal implementation. > > i only skimmed the first part of this thread, but assumed if you're > talking about performance, you'd want to compare the overall impact on > your system(s) of Net::SMTP vs. |qmail-inject. you cannot measure the > overall impact just using Benchmark.pm is all i'm trying to clarify. A better plan for such systems is to have a queue in your database for parameters for e-mails to send. Insert a record in the database and let your web server continue processing. Have a separate process possibly on a separate machine or possibly on multiple machines do selects from that queue and deliver mail. I think the fastest way is over a single SMTP connection to the mail relay rather than forking a process to inject the mail. This keeps the very variable -- even on your own systems -- mail latency completely out of the critical path for web server requests. Which is really the key measure that dictates the requests/s you can serve. -- greg
RE: open(FH,'|qmail-inject') fails
If you are using qmail-inject in a high volume configuration, make sure you reduce or eliminate most of its logging, otherwise the system will spin its wheels doing logging work and not sending out emails. Haven't compared Net::SMTP to qmail-inject, but qmail-inject is significantly faster than straight sendmail on Linux. Jeff -Original Message- From: Doug MacEachern [mailto:[EMAIL PROTECTED]] Sent: Monday, September 25, 2000 5:21 PM To: Stas Bekman Cc: Bill Moseley; Modperl Subject: Re: open(FH,'|qmail-inject') fails On Mon, 25 Sep 2000, Stas Bekman wrote: > All you care about is to measure the time between email sending start and > end (when the process continues on its execution flow). Why should one > care about the details of internal implementation. i only skimmed the first part of this thread, but assumed if you're talking about performance, you'd want to compare the overall impact on your system(s) of Net::SMTP vs. |qmail-inject. you cannot measure the overall impact just using Benchmark.pm is all i'm trying to clarify.
Re: open(FH,'|qmail-inject') fails
On Mon, 25 Sep 2000, Matt Sergeant wrote: > On Mon, 25 Sep 2000, Stas Bekman wrote: > > On Mon, 25 Sep 2000, Doug MacEachern wrote: > > > On Mon, 25 Sep 2000, Stas Bekman wrote: > > > > All you care about is to measure the time between email sending start and > > > > end (when the process continues on its execution flow). Why should one > > > > care about the details of internal implementation. > > > i only skimmed the first part of this thread, but assumed if you're > > > talking about performance, you'd want to compare the overall impact on > > > your system(s) of Net::SMTP vs. |qmail-inject. you cannot measure the > > > overall impact just using Benchmark.pm is all i'm trying to clarify. > > Yup, you are right. The overall impact is important as well. > > * Benchmark to see the responce times. > > * Send lots of emails and watch to compare the overall impact. > And unfortunately you'll also miss certain issues with that too, unless > its a live system, for example DNS lookup issues with sending to different > hosts. The joys of benchmarking code when the network is involved! Plus the minor fact that email systems seem highly non-deterministic in general :) We have some code for London Transport, which seems to be very random about when it actually sends the email. MBM -- Tell me, O Octopus, I begs, / Is those things arms, or is they legs? / I marvel at thee, Octopus; / If I were thou, I'd call me us. -- Ogden Nash
Re: open(FH,'|qmail-inject') fails
On Mon, 25 Sep 2000, Matt Sergeant wrote: > On Mon, 25 Sep 2000, Stas Bekman wrote: > > > On Mon, 25 Sep 2000, Doug MacEachern wrote: > > > > > On Mon, 25 Sep 2000, Stas Bekman wrote: > > > > > > > All you care about is to measure the time between email sending start and > > > > end (when the process continues on its execution flow). Why should one > > > > care about the details of internal implementation. > > > > > > i only skimmed the first part of this thread, but assumed if you're > > > talking about performance, you'd want to compare the overall impact on > > > your system(s) of Net::SMTP vs. |qmail-inject. you cannot measure the > > > overall impact just using Benchmark.pm is all i'm trying to clarify. > > > > Yup, you are right. The overall impact is important as well. > > > > * Benchmark to see the responce times. > > > > * Send lots of emails and watch to compare the overall impact. > > And unfortunately you'll also miss certain issues with that too, unless > its a live system, for example DNS lookup issues with sending to different > hosts. The joys of benchmarking code when the network is involved! Now I understand why do I get so much SPAM. People use it to test the network implication on the email sending. And since I get quite a lot of identical SPAM, it looks like it takes them a while to figure out the exact figures. So after all, I'm contributing to someone else by receiving all this SPAM. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: open(FH,'|qmail-inject') fails
On Mon, 25 Sep 2000, Stas Bekman wrote: > On Mon, 25 Sep 2000, Doug MacEachern wrote: > > > On Mon, 25 Sep 2000, Stas Bekman wrote: > > > > > All you care about is to measure the time between email sending start and > > > end (when the process continues on its execution flow). Why should one > > > care about the details of internal implementation. > > > > i only skimmed the first part of this thread, but assumed if you're > > talking about performance, you'd want to compare the overall impact on > > your system(s) of Net::SMTP vs. |qmail-inject. you cannot measure the > > overall impact just using Benchmark.pm is all i'm trying to clarify. > > Yup, you are right. The overall impact is important as well. > > * Benchmark to see the responce times. > > * Send lots of emails and watch to compare the overall impact. And unfortunately you'll also miss certain issues with that too, unless its a live system, for example DNS lookup issues with sending to different hosts. The joys of benchmarking code when the network is involved! -- Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org | AxKit: http://axkit.org
Re: open(FH,'|qmail-inject') fails
On Mon, 25 Sep 2000, Doug MacEachern wrote: > On Mon, 25 Sep 2000, Stas Bekman wrote: > > > All you care about is to measure the time between email sending start and > > end (when the process continues on its execution flow). Why should one > > care about the details of internal implementation. > > i only skimmed the first part of this thread, but assumed if you're > talking about performance, you'd want to compare the overall impact on > your system(s) of Net::SMTP vs. |qmail-inject. you cannot measure the > overall impact just using Benchmark.pm is all i'm trying to clarify. Yup, you are right. The overall impact is important as well. * Benchmark to see the responce times. * Send lots of emails and watch to compare the overall impact. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: open(FH,'|qmail-inject') fails
On Mon, 25 Sep 2000, Stas Bekman wrote: > All you care about is to measure the time between email sending start and > end (when the process continues on its execution flow). Why should one > care about the details of internal implementation. i only skimmed the first part of this thread, but assumed if you're talking about performance, you'd want to compare the overall impact on your system(s) of Net::SMTP vs. |qmail-inject. you cannot measure the overall impact just using Benchmark.pm is all i'm trying to clarify.
Re: open(FH,'|qmail-inject') fails
On Mon, 25 Sep 2000, Doug MacEachern wrote: > On Fri, 8 Sep 2000, Stas Bekman wrote: > > > As Perrin has suggested, benchmark it an see what's faster. It's so > > simple. > > come on stas, benchmarking Net::SMTP vs. pipe to a program is nothing > close to simple, it is not something you can use Benchmark.pm to cover all > the bases (e.g. tcpserver forking, what happens after qmail-inject exits, > etc.) All you care about is to measure the time between email sending start and end (when the process continues on its execution flow). Why should one care about the details of internal implementation. At least that's what I meant by saying "benchmark it". _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: open(FH,'|qmail-inject') fails
On Fri, 8 Sep 2000, Stas Bekman wrote: > As Perrin has suggested, benchmark it an see what's faster. It's so > simple. come on stas, benchmarking Net::SMTP vs. pipe to a program is nothing close to simple, it is not something you can use Benchmark.pm to cover all the bases (e.g. tcpserver forking, what happens after qmail-inject exits, etc.)
Re: open(FH,'|qmail-inject') fails
On Thu, Sep 07, 2000 at 04:56:59PM -0400, Roger Espel Llima wrote: > On Thu, Sep 07, 2000 at 01:25:21PM -0700, Randal L. Schwartz wrote: > > Man, if I see ONE MORE script that checks for a "legal email", > > well, you could always try to check the address against rfc822... but > that would be one hell of a regexp, and it'd be mostly useless, since > the worst that can happen is a bounced email. Well, see Jeff Friedl's RFC822 REx in MRE, and more interestingly Abigail's RFC::RFC822::Address, which uses Parse::RecDescent. Of course, as you say, in that instance there's not a whole lot of point to the check. But Abigail's module is great fun nonetheless. -dlc
Re: [OT] Re: open(FH,'|qmail-inject') fails
> "Bill" == Bill Moseley <[EMAIL PROTECTED]> writes: Bill> Did you see this of Abigail's? Bill> http://x58.deja.com/getdoc.xp?AN=531471039 Yeah, that's already in the CPAN at RFC::RFC822::Address, just a different way of doing it from Email::Valid. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 <[EMAIL PROTECTED]> http://www.stonehenge.com/merlyn/> Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: [OT] Re: open(FH,'|qmail-inject') fails
At 04:39 PM 09/11/00 +0200, Alexander Farber (EED) wrote: >"Randal L. Schwartz" wrote: >> Man, if I see ONE MORE script that checks for a "legal email", >> I'm gonna scream. Matter of fact, I already did. :) > >I screamed when I've seen the correct version too :-) It is at >http://public.yahoo.com/~jfriedl/regex/code.html and the regex >for URL's is at http://foad.org/~abigail/Perl/url2.html Did you see this of Abigail's? http://x58.deja.com/getdoc.xp?AN=531471039 Bill Moseley mailto:[EMAIL PROTECTED]
[OT] Re: open(FH,'|qmail-inject') fails
"Randal L. Schwartz" wrote: > Man, if I see ONE MORE script that checks for a "legal email", > I'm gonna scream. Matter of fact, I already did. :) I screamed when I've seen the correct version too :-) It is at http://public.yahoo.com/~jfriedl/regex/code.html and the regex for URL's is at http://foad.org/~abigail/Perl/url2.html
Re: open(FH,'|qmail-inject') fails
Perrin Harkins wrote: > > On Fri, 8 Sep 2000, Stas Bekman wrote: > > > As far as benchmarks are concerned, I'm sending one mail after having > > > displayed the page, so it shoul'dnt matter much ... > > > > Yeah, and everytime you get 1M process fired up... > > Nevertheless, in benchmarks we ran we found forking qmail-inject to be > quite a bit faster than Net::SMTP. I'd say that at least from a > command-line script qmail-inject is a more scalable approach. Or even better would be to use qmail-remote and avoid the queue, but thats a qmail and not a mod_perl thing. BTW I'm looking at this at the mo and have decided to go back to a dump email to db -> crond robot sends mail to port 25 -> mail delt with there. Why ? Well this is the most scalable, and its cross platform, talking directly via pipes / forks etc usually has a limit somewhere, and all too often I may reach it far sooner than I thought. Greg > > - Perrin
Re: open(FH,'|qmail-inject') fails
On Sat, Sep 09, 2000 at 12:29:36PM -0700, Bill Moseley wrote: > At 11:15 PM 09/08/00 +0200, Stas Bekman wrote: > >On Fri, 8 Sep 2000, Bill Moseley wrote: > >> I just looked at my old mail sending module a few days ago that uses > >> sendmail and would fallback to Net::SMTP if sendmail wasn't available (it > >> was running on Win at one point, argh!). I just removed the Net::SMTP > >> part. Are you saying that I removed the wrong code? > > >> At 10:31 AM 09/08/00 +0200, Stas Bekman wrote: > > > >As Perrin has suggested, benchmark it an see what's faster. It's so > >simple. > > > > Benchmark: timing 1 iterations of SMTP, Sendmail... > SMTP: 60 wallclock secs ( 0.03 usr + 0.00 sys = 0.03 CPU) @ 33.33/s > (n=1) > (warning: too few iterations for a reliable count) > Sendmail: 0 wallclock secs ( 0.00 usr 0.00 sys + 0.01 cusr 0.01 csys = > 0.02 CPU) > (warning: too few iterations for a reliable count) > > I guess it isn't a fair test ;) I had my DNS servers unplugged and used > -odd with sendmail. And I guess if I set a timeout of 60 seconds on > Net::SMTP I better be ready to wait that long. > > > > Seriously, though, I can't imagine how to benchmark such a complicated > thing as sending mail. I can do it on my machine, I guess, but that > doesn't tell me how it will work on the live machine that's also sending > out a mailing list or running a faulty nscd on Solaris that hangs DNS > requests, or mailing to [EMAIL PROTECTED] and DNS is hopping from DNS to > DNS to do the lookup. > > The nscd thing is the reason I moved from using Net::SMTP; SMTP worked > great most of the time, but really messed things up when the DNS was not > working. Forking may or may not be slower, but it like more of a sure bet. > > Have fun, > > > > > Bill Moseley > mailto:[EMAIL PROTECTED] All your objections about smtp will be gone if you set up a dedicated local account just for processing emails from modperl. Since the account is local, Net::SMTP will get done instantly and the modperl can get back doing what it is supposed to be doing instead of waiting for dns resolution, nscd and what not. So in modperl, you write the real email as the body of email to the local account, giving it a special subject line such as "send out the body text". Then you set up procmail or .forward for that local account so that it will do "sendmail -oi -t" or whatever to actually send the body of text. It can take as much time as it wants without interfering the website service. Richard
Re: open(FH,'|qmail-inject') fails
At 11:15 PM 09/08/00 +0200, Stas Bekman wrote: >On Fri, 8 Sep 2000, Bill Moseley wrote: >> I just looked at my old mail sending module a few days ago that uses >> sendmail and would fallback to Net::SMTP if sendmail wasn't available (it >> was running on Win at one point, argh!). I just removed the Net::SMTP >> part. Are you saying that I removed the wrong code? >> At 10:31 AM 09/08/00 +0200, Stas Bekman wrote: > >As Perrin has suggested, benchmark it an see what's faster. It's so >simple. Benchmark: timing 1 iterations of SMTP, Sendmail... SMTP: 60 wallclock secs ( 0.03 usr + 0.00 sys = 0.03 CPU) @ 33.33/s (n=1) (warning: too few iterations for a reliable count) Sendmail: 0 wallclock secs ( 0.00 usr 0.00 sys + 0.01 cusr 0.01 csys = 0.02 CPU) (warning: too few iterations for a reliable count) I guess it isn't a fair test ;) I had my DNS servers unplugged and used -odd with sendmail. And I guess if I set a timeout of 60 seconds on Net::SMTP I better be ready to wait that long. Seriously, though, I can't imagine how to benchmark such a complicated thing as sending mail. I can do it on my machine, I guess, but that doesn't tell me how it will work on the live machine that's also sending out a mailing list or running a faulty nscd on Solaris that hangs DNS requests, or mailing to [EMAIL PROTECTED] and DNS is hopping from DNS to DNS to do the lookup. The nscd thing is the reason I moved from using Net::SMTP; SMTP worked great most of the time, but really messed things up when the DNS was not working. Forking may or may not be slower, but it like more of a sure bet. Have fun, Bill Moseley mailto:[EMAIL PROTECTED]
RE: open(FH,'|qmail-inject') fails
Title: RE: open(FH,'|qmail-inject') fails Another approach to is to write the email directly into the queue. I've used this approach and it's very fast. After you write your email to the qmail queue, you write a value of 1 to a named pipe that qmail reads off of. This causes a qmail process (there are like 20 different ones and I forget which is which - check the docs) to wake up and drain the queue. If you want anymore speed then that, you have to either install ram disks or seriously write your own mta. We installed ram disks =) Shane -Original Message- From: Andrew Dunstan [mailto:[EMAIL PROTECTED]] Sent: Friday, September 08, 2000 9:52 PM To: Modperl Subject: Re: open(FH,'|qmail-inject') fails Regarding cost of forking etc.: Your mileage will undoubtedly vary, according to OS and MTA. Last time I did work on this was about a year ago on Solaris 2.6, with sendmail and postfix. In both cases using Net::SMTP was far faster. IIRC, with postfix there is no forking cost at all, as its daemon does not fork on connect (it uses a select() loop instead). Talking on the SMTP port is actually Wietse Venema's recommended method for fastest injection into the postfix queue. It also has the advantage over other methods that it is totally MTA independent. andrew
Re: open(FH,'|qmail-inject') fails
Regarding cost of forking etc.: Your mileage will undoubtedly vary, according to OS and MTA. Last time I did work on this was about a year ago on Solaris 2.6, with sendmail and postfix. In both cases using Net::SMTP was far faster. IIRC, with postfix there is no forking cost at all, as its daemon does not fork on connect (it uses a select() loop instead). Talking on the SMTP port is actually Wietse Venema's recommended method for fastest injection into the postfix queue. It also has the advantage over other methods that it is totally MTA independent. andrew
RE: open(FH,'|qmail-inject') fails
Stas wrote: > On Fri, 8 Sep 2000, Perrin Harkins wrote: > > Nevertheless, in benchmarks we ran we found forking qmail-inject to be > > quite a bit faster than Net::SMTP. I'd say that at least from a > > command-line script qmail-inject is a more scalable approach. > > Quite possible, I was talking about the fat sendmail binaries :) Yes, quite possible. Using SMTP and qmail-inject both have the overhead of a fork, because the SMTP tcpserver will fork off a copy of qmail-smtpd to handle the request. Additionally, the SMTP tcpserver is probably doing a reverse DNS lookup and probably an ident lookup which would probably cause another fork for identd. (Both reverse DNS and ident lookup are enabled by default in ucspi-tcp-0.84.) This network activity and possibly another fork will cause delay. The overhead of forking directly off from mod_perl does not seem so bad when you look at copy-on-write memory managers. The fork: (a) does not cause copying of the big mod_perl process at fork thanks to copy-on-write, and (b) there will be virtually no dirtying of pages and copying because a exec() will be immediately done. A possible problem is qmail-inject inheriting a bunch of filehandles from mod_perl, but they should all be marked close-on-exec. David Harris President, DRH Internet Inc. [EMAIL PROTECTED] http://www.drh.net/
Re: open(FH,'|qmail-inject') fails
On Fri, 8 Sep 2000, Bill Moseley wrote: > At 10:31 AM 09/08/00 +0200, Stas Bekman wrote: > >Net::SMTP works perfectly and doesn't lack any documentation. If there is > >a bunch of folks who use mod_perl for their guestbook sites it's perfectly > >Ok to run sendmail/postfix/whatever program you like... But it just > >doesn't scale for the big projects. > > > >For a simple replacement of sendmail routine with Net::SMTP grab: > >http://www.stason.org/works/scripts/mail-lib.pl > > I guess I need some help understanding this, as it seems I have things > backwards. > > I don't use Net::SMTP as I feel I'm at the whim of the SMTP server > (possibly on another machine) -- at least forking and asking sendmail to > queue the file, although eating memory and requiring a fork, seems like a > constant. This is probably all hogwash, but I like constants. Net::SMTP is just an interface to your mail (SMTP) server, which makes Net::SMTP a client. When you fire sendmail or an equivalent you run a client. It's how you configure the mailserver what makes the difference. In case of sendmail it's both a client and server of course. I'm not expert in mailing servers, so I guess someone on the list may provide the necessary details of making the server queue the mails. > I just looked at my old mail sending module a few days ago that uses > sendmail and would fallback to Net::SMTP if sendmail wasn't available (it > was running on Win at one point, argh!). I just removed the Net::SMTP > part. Are you saying that I removed the wrong code? As Perrin has suggested, benchmark it an see what's faster. It's so simple. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: open(FH,'|qmail-inject') fails
On Fri, 8 Sep 2000, Perrin Harkins wrote: > On Fri, 8 Sep 2000, Stas Bekman wrote: > > > As far as benchmarks are concerned, I'm sending one mail after having > > > displayed the page, so it shoul'dnt matter much ... > > > > Yeah, and everytime you get 1M process fired up... > > Nevertheless, in benchmarks we ran we found forking qmail-inject to be > quite a bit faster than Net::SMTP. I'd say that at least from a > command-line script qmail-inject is a more scalable approach. Quite possible, I was talking about the fat sendmail binaries :) _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: open(FH,'|qmail-inject') fails
> "Bill" == Bill Moseley <[EMAIL PROTECTED]> writes: Bill> I wouldn't want to depend on sending a lot of mail to a mail server I Bill> didn't have control over in the middle of a request. Unless the mail is for very local delivery, EVERY piece of mail goes to a mail server that you don't have control over in the middle of the request. :-) -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 <[EMAIL PROTECTED]> http://www.stonehenge.com/merlyn/> Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: open(FH,'|qmail-inject') fails
On Fri, 8 Sep 2000 [EMAIL PROTECTED] wrote: > There is a very important reason for having to fork qmail-inject. Qmail > by default will not allow mail relaying as a good security measure. You > don't want your mail server to be used for spamming especially if you > have a T3 or a T1 link. Anyone who is allowing sendmail to relay is in > for trouble - there has been occassions when people have been sued just > because the spamming originated from their servers. > > You cant use Net::SMTP because when you try to send out emails to other > domains by connecting to the localhost Qmail will reject it because the > reciepient is not defined as a local user or a virtual domain. You add these lines to /etc/hosts.allow and sleep peacefully: tcp-env: 192.168. : setenv = RELAYCLIENT tcp-env: 10. : setenv = RELAYCLIENT tcp-env: : setenv = RELAYCLIENT tcp-env: 127.0.0.1 : setenv = RELAYCLIENT (the last one there might not be necessary). Oh, and ensure your firewall doesn't allow incoming users pretending to come from 10.* or 192.168.* -- Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org | AxKit: http://axkit.org
Re: open(FH,'|qmail-inject') fails
On Fri, 8 Sep 2000, Stas Bekman wrote: > > As far as benchmarks are concerned, I'm sending one mail after having > > displayed the page, so it shoul'dnt matter much ... > > Yeah, and everytime you get 1M process fired up... Nevertheless, in benchmarks we ran we found forking qmail-inject to be quite a bit faster than Net::SMTP. I'd say that at least from a command-line script qmail-inject is a more scalable approach. - Perrin
Re: open(FH,'|qmail-inject') fails
At 10:07 AM 09/08/00 -0700, brian moseley wrote: >On Fri, 8 Sep 2000, Bill Moseley wrote: > >> I don't know how well either of these scale. But if >> scaling is important I'd think it best not to rely on >> some smtp daemon. > >this is a joke, right? > >'i want to send lots of mail, i better not use a MAIL >SERVER'. 'some' was the operative word. Sorry if that was unclear. I wouldn't want to depend on sending a lot of mail to a mail server I didn't have control over in the middle of a request. Bill Moseley mailto:[EMAIL PROTECTED]
Re: open(FH,'|qmail-inject') fails
On Fri, Sep 08, 2000 at 11:17:31AM -0400, [EMAIL PROTECTED] wrote: > I have an immense amount of respect for you Randal, but I think you're > generalizing a bit much here. There are a number of cases where checking > an email address' validity makes perfectly good sense. The most obvious > is just plain human-computer interface design. If I can give the user a > message "hay, that's not a valid email address" instead of them wondering > why they never received an email, that's makes the interaction more > intuitive. Many, many scripts end up calling sendmail to send mail. I'm > not for a moment going to applaud that method, but it does mean that shell > escapes in an email address will cause problems. If you use sendmail's -t option to let it read the mail addresses from the message itself you do not need to pass email addresses on command lines. This is much more secure and relies instead on sendmail's rather largish machinery to parse email adresses. -- Jens-Uwe Mager HELIOS Software GmbH Steinriede 3 30827 Garbsen Germany Phone: +49 5131 709320 FAX:+49 5131 709325 Internet: [EMAIL PROTECTED]
Re: open(FH,'|qmail-inject') fails
On Fri, 8 Sep 2000, Bill Moseley wrote: > I don't know how well either of these scale. But if > scaling is important I'd think it best not to rely on > some smtp daemon. this is a joke, right? 'i want to send lots of mail, i better not use a MAIL SERVER'.
Re: open(FH,'|qmail-inject') fails
At 10:31 AM 09/08/00 +0200, Stas Bekman wrote: >Net::SMTP works perfectly and doesn't lack any documentation. If there is >a bunch of folks who use mod_perl for their guestbook sites it's perfectly >Ok to run sendmail/postfix/whatever program you like... But it just >doesn't scale for the big projects. > >For a simple replacement of sendmail routine with Net::SMTP grab: >http://www.stason.org/works/scripts/mail-lib.pl I guess I need some help understanding this, as it seems I have things backwards. I don't use Net::SMTP as I feel I'm at the whim of the SMTP server (possibly on another machine) -- at least forking and asking sendmail to queue the file, although eating memory and requiring a fork, seems like a constant. This is probably all hogwash, but I like constants. So for times when I send mail just once in a while I fork & exec sendmail. We got 2G of RAM for a reason. But for times when I need to send mail on many requests I write to a queue file and use cron to process the mail. I don't know how well either of these scale. But if scaling is important I'd think it best not to rely on some smtp daemon. I just looked at my old mail sending module a few days ago that uses sendmail and would fallback to Net::SMTP if sendmail wasn't available (it was running on Win at one point, argh!). I just removed the Net::SMTP part. Are you saying that I removed the wrong code? BTW -- I just looked and I have some other places where I use open3 to open sendmail. I have a note about sendmail generating an error by writing to STDERR or STDOUT only and not returning failure. This sound familiar to anyone? Bill Moseley mailto:[EMAIL PROTECTED]
Re: open(FH,'|qmail-inject') fails
There is a very important reason for having to fork qmail-inject. Qmail by default will not allow mail relaying as a good security measure. You don't want your mail server to be used for spamming especially if you have a T3 or a T1 link. Anyone who is allowing sendmail to relay is in for trouble - there has been occassions when people have been sued just because the spamming originated from their servers. You cant use Net::SMTP because when you try to send out emails to other domains by connecting to the localhost Qmail will reject it because the reciepient is not defined as a local user or a virtual domain. In anycase when one writes a email and sends it the email client doesnt go and deliver it to the SMTP port (atleast if they are on the same server). Rather the email client executes the MDA and allows it to handle the situation appripriately. In case of qmail it is /var/qmail/bin/qmail-inject. I personally don't like forking from modperl but that doesn't imply calling qmail-inject is a wrong approach either. I am sure their are work arounds for most situation but sending a email with users submission is a very quick and dirty solution if you are hard pressed for time. Mithun Andrew Dunstan wrote: > > Could someone please explain to me why everybody seems so intent on having a >mod_perl handler fork in order to send mail? Why not just use the very common >Net::SMTP package which just talks on an SMTP socket to whatever mailhost you have >(localhost or other). There are other packages on CPAN which perhaps have more power, >but still don't fork, if that's what you need. Every benchmark I've done (quite a few >;-) shows that this is far faster way of sending mail. > > My understanding (correct me if I'm wrong) is that in general having a mod_perl >handler fork is a Bad Thing (tm). > > (and of course there is even less danger with funky email addresses with shell >metacharacters that way, too) > > I recall with some fondness Randal's "useless use of cat" awards - maybe we need to >create a "useless use of fork" award :-) > > cheers > > andrew
Re: open(FH,'|qmail-inject') fails
On 7 Sep 2000 Randal L. Schwartz wrote: > This is neither necessary nor sufficient. Please stop with this > nonsense. An email address can have ANY CHARACTER OF THE PRINTABLE > ASCII SEQUENCE. An email address NEVER NEEDS TO GET NEAR A SHELL, so > ALL CHARACTERS ARE SAFE. Clear? Man, if I see ONE MORE script that > checks for a "legal email", I'm gonna scream. Matter of fact, I > already did. :) I have an immense amount of respect for you Randal, but I think you're generalizing a bit much here. There are a number of cases where checking an email address' validity makes perfectly good sense. The most obvious is just plain human-computer interface design. If I can give the user a message "hay, that's not a valid email address" instead of them wondering why they never received an email, that's makes the interaction more intuitive. Many, many scripts end up calling sendmail to send mail. I'm not for a moment going to applaud that method, but it does mean that shell escapes in an email address will cause problems. -- If you're not part of the solution, you're part of the precipitate. - Steven Wright
Re: open(FH,'|qmail-inject') fails
On Fri, 8 Sep 2000, Nicolas MONNET wrote: > On Thu, 7 Sep 2000, Andrew Dunstan wrote: > > |Could someone please explain to me why everybody seems so intent on > |having a mod_perl handler fork in order to send mail? Why not just use > |the very common Net::SMTP package which just talks on an SMTP socket to > |whatever mailhost you have (localhost or other). There are other > |packages on CPAN which perhaps have more power, but still don't fork, if > |that's what you need. Every benchmark I've done (quite a few ;-) shows > |that this is far faster way of sending mail. > > To answer your question, I have tried some times ago, with one such module > (not sure if it was this one) and as the documentation was lacking I had > no clear idea of what was going on and could'nt get it to work properly. Net::SMTP works perfectly and doesn't lack any documentation. If there is a bunch of folks who use mod_perl for their guestbook sites it's perfectly Ok to run sendmail/postfix/whatever program you like... But it just doesn't scale for the big projects. For a simple replacement of sendmail routine with Net::SMTP grab: http://www.stason.org/works/scripts/mail-lib.pl Why you see only posts which talk about forking the external process? Because all those who use Net::SMTP don't have any problems, therefore they are all quiet :) > As far as benchmarks are concerned, I'm sending one mail after having > displayed the page, so it shoul'dnt matter much ... Yeah, and everytime you get 1M process fired up... > |My understanding (correct me if I'm wrong) is that in general having a > |mod_perl handler fork is a Bad Thing (tm). > > Dunno about this; never read anything against it. A lame excuse :) You know, not knowing the law, doesn't wave away a responsibility for your deeds... . Just to tell you that there is *something* to read against it: http://perl.apache.org/guide/performance.html#Forking_and_Executing_Subprocess > |(and of course there is even less danger with funky email addresses > with |shell metacharacters that way, too) > > I pay attention to that kind of issues; as a matter of fact, the command I > use is: open(OUTP,'|/path/qmail-inject -H') -- so no variable in the > shell, and print the recipient in the output. This issue is explained very well at: http://perl.apache.org/guide/performance.html#Executing_system_in_the_Right_ No metachars issue anymore. P.S. I'm not responding to this email in order to flame Nicolas, but to correct things that otherwise might be assumes as correct for those among us with less experience. Whatever you see posted on the list should never be taken for granted since we all make mistakes, even the experts among us. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: open(FH,'|qmail-inject') fails
On Thu, 7 Sep 2000, Andrew Dunstan wrote: |Could someone please explain to me why everybody seems so intent on |having a mod_perl handler fork in order to send mail? Why not just use |the very common Net::SMTP package which just talks on an SMTP socket to |whatever mailhost you have (localhost or other). There are other |packages on CPAN which perhaps have more power, but still don't fork, if |that's what you need. Every benchmark I've done (quite a few ;-) shows |that this is far faster way of sending mail. To answer your question, I have tried some times ago, with one such module (not sure if it was this one) and as the documentation was lacking I had no clear idea of what was going on and could'nt get it to work properly. As far as benchmarks are concerned, I'm sending one mail after having displayed the page, so it shoul'dnt matter much ... |My understanding (correct me if I'm wrong) is that in general having a |mod_perl handler fork is a Bad Thing (tm). Dunno about this; never read anything against it. |(and of course there is even less danger with funky email addresses with |shell metacharacters that way, too) I pay attention to that kind of issues; as a matter of fact, the command I use is: open(OUTP,'|/path/qmail-inject -H') -- so no variable in the shell, and print the recipient in the output. |I recall with some fondness Randal's "useless use of cat" awards - maybe |we need to create a "useless use of fork" award :-)
Re: open(FH,'|qmail-inject') fails
Could someone please explain to me why everybody seems so intent on having a mod_perl handler fork in order to send mail? Why not just use the very common Net::SMTP package which just talks on an SMTP socket to whatever mailhost you have (localhost or other). There are other packages on CPAN which perhaps have more power, but still don't fork, if that's what you need. Every benchmark I've done (quite a few ;-) shows that this is far faster way of sending mail. My understanding (correct me if I'm wrong) is that in general having a mod_perl handler fork is a Bad Thing (tm). (and of course there is even less danger with funky email addresses with shell metacharacters that way, too) I recall with some fondness Randal's "useless use of cat" awards - maybe we need to create a "useless use of fork" award :-) cheers andrew
Re: open(FH,'|qmail-inject') fails
On Thu, Sep 07, 2000 at 01:25:21PM -0700, Randal L. Schwartz wrote:
> > "Roger" == Roger Espel Llima <[EMAIL PROTECTED]> writes:
>
> Roger> # limit allowed characters in email addresses
> Roger> $to =~ tr/-a-zA-Z0-9_+%$.,:!@=()[]//cd;
>
> * An email address can have ANY CHARACTER OF THE PRINTABLE ASCII SEQUENCE.
> * An email address NEVER NEEDS TO GET NEAR A SHELL, so ALL CHARACTERS
> * ARE SAFE.
err, you're right. my code had another bug too: qmail-inject's -f
option wants the email address only, not the "Full Name "
thing. that's what I get for trying to make "cosmetic" changes to code
just before posting.
here's the function again, exactly as we use it, and without the
filtering:
sub sendmail {
my ($from, $to, $subject, $message, $name) = @_;
local *MAIL;
# safe pipes adapted from man perlipc
local $SIG{PIPE} = sub {};
my $pid = open (MAIL, "|-");
local $SIG{ALRM} = sub { CORE::exit; };
unless (defined $pid) {
# fork failed!
return "Error sending mail.";
}
if ($pid) {
# parent
if (defined $name) { print MAIL "From: $name <$from>\n"; }
else { print MAIL "From: $from\n"; }
print MAIL "To: $to\n";
print MAIL "Subject: $subject\n\n";
print MAIL $message, "\n";
close MAIL;
return 1;
} else {
# child
exec("/var/qmail/bin/qmail-inject", "-f", $from, "--", $to)
|| CORE::exit;
}
}
> Man, if I see ONE MORE script that checks for a "legal email",
well, you could always try to check the address against rfc822... but
that would be one hell of a regexp, and it'd be mostly useless, since
the worst that can happen is a bounced email.
> I'm gonna scream. Matter of fact, I already did. :)
feels better doesn't it? :)
--
Roger Espel Llima, [EMAIL PROTECTED]
http://www.iagora.com/~espel/index.html
Re: open(FH,'|qmail-inject') fails
> "Roger" == Roger Espel Llima <[EMAIL PROTECTED]> writes: Roger> # limit allowed characters in email addresses Roger> $to =~ tr/-a-zA-Z0-9_+%$.,:!@=()[]//cd; This is neither necessary nor sufficient. Please stop with this nonsense. ** * * An email address can have ANY CHARACTER OF THE PRINTABLE ASCII SEQUENCE. * * An email address NEVER NEEDS TO GET NEAR A SHELL, so ALL CHARACTERS * ARE SAFE. * ** Clear? Man, if I see ONE MORE script that checks for a "legal email", I'm gonna scream. Matter of fact, I already did. :) -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 <[EMAIL PROTECTED]> http://www.stonehenge.com/merlyn/> Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: open(FH,'|qmail-inject') fails
On Thu, 7 Sep 2000, Roger Espel Llima wrote:
> > > Might be a faq, but why would open(FH,'|qmail-inject') fail with
> > > fatal: read-error from within mod_perl?
> > Use
> > open MAIL, "| /var/qmail/bin/qmail-inject" or &die_html("test");
> > print MAIL "[your mail]";
> > close MAIL;
> >
> > I suppose you forgot the full path to qmail-inject ...
>
> I use this, it's a bit more complete and paranoid:
>
> # send mail from the system to someone
> sub sendmail {
> my ($from, $to, $subject, $message, $name) = @_;
> local *MAIL;
>
> # limit allowed characters in email addresses
> $to =~ tr/-a-zA-Z0-9_+%$.,:!@=()[]//cd;
>
> # safe pipes adapted from man perlipc
> local $SIG{PIPE} = sub {};
> my $pid = open (MAIL, "|-");
> local $SIG{ALRM} = sub { CORE::exit; };
>
> unless (defined $pid) {
> # fork failed!
> return "Error sending mail.";
> }
>
> if ($pid) {
> # parent
> $from = "$name <$from>" if defined $name;
> print MAIL "From: $from\n";
> print MAIL "To: $to\n";
> print MAIL "Subject: $subject\n\n";
> print MAIL $message, "\n";
> close MAIL;
> return 1;
> } else {
> # child
> exec("/var/qmail/bin/qmail-inject", "-f", $from, "--", $to)
> || CORE::exit;
> }
> }
>
> --
> Roger Espel Llima, [EMAIL PROTECTED]
> http://www.iagora.com/~espel/index.html
>
Just a bit of a note, the $pid will come back even if the pipe being
opened to qmail-inject fails. A child gets forked off to handle opening
the pipe, and the process id comes back from that, not qmail-inject.
You should also check this:
close(MAIL) or die "Couldn't open pipe to qmail-inject";
(I learned this one the hard way).
Brian Nilsen
[EMAIL PROTECTED]
Re: open(FH,'|qmail-inject') fails
> > Might be a faq, but why would open(FH,'|qmail-inject') fail with
> > fatal: read-error from within mod_perl?
> Use
> open MAIL, "| /var/qmail/bin/qmail-inject" or &die_html("test");
> print MAIL "[your mail]";
> close MAIL;
>
> I suppose you forgot the full path to qmail-inject ...
I use this, it's a bit more complete and paranoid:
# send mail from the system to someone
sub sendmail {
my ($from, $to, $subject, $message, $name) = @_;
local *MAIL;
# limit allowed characters in email addresses
$to =~ tr/-a-zA-Z0-9_+%$.,:!@=()[]//cd;
# safe pipes adapted from man perlipc
local $SIG{PIPE} = sub {};
my $pid = open (MAIL, "|-");
local $SIG{ALRM} = sub { CORE::exit; };
unless (defined $pid) {
# fork failed!
return "Error sending mail.";
}
if ($pid) {
# parent
$from = "$name <$from>" if defined $name;
print MAIL "From: $from\n";
print MAIL "To: $to\n";
print MAIL "Subject: $subject\n\n";
print MAIL $message, "\n";
close MAIL;
return 1;
} else {
# child
exec("/var/qmail/bin/qmail-inject", "-f", $from, "--", $to)
|| CORE::exit;
}
}
--
Roger Espel Llima, [EMAIL PROTECTED]
http://www.iagora.com/~espel/index.html
Re: open(FH,'|qmail-inject') fails
Thanks a lot, seems to be it, never heard about that env var. That's one weird behavior. On 7 Sep 2000, Frank D. Cringle wrote: |Nicolas MONNET <[EMAIL PROTECTED]> writes: |> Might be a faq, but why would open(FH,'|qmail-inject') fail with |> fatal: read-error from within mod_perl? | |Are the files in /var/qmail/control world readable? |Is QMAILMFTFILE defined in the environment and pointing to a file that |the httpd process cannot read?
Re: open(FH,'|qmail-inject') fails
Nicolas MONNET <[EMAIL PROTECTED]> writes: > Might be a faq, but why would open(FH,'|qmail-inject') fail with > fatal: read-error from within mod_perl? Are the files in /var/qmail/control world readable? Is QMAILMFTFILE defined in the environment and pointing to a file that the httpd process cannot read? -- Frank Cringle, [EMAIL PROTECTED] voice: (+49 7745) 928759; fax: 928761
Re: open(FH,'|qmail-inject') fails
No, no the full path is there, I just did'nt copy it.
On Thu, 7 Sep 2000, Frédéric Schwien wrote:
|Date: Thu, 7 Sep 2000 12:18:20 +0200
|From: Frédéric Schwien <[EMAIL PROTECTED]>
|To: Nicolas MONNET <[EMAIL PROTECTED]>, [EMAIL PROTECTED]
|Subject: Re: open(FH,'|qmail-inject') fails
|
|> Might be a faq, but why would open(FH,'|qmail-inject') fail with
|> fatal: read-error from within mod_perl?
|Use
|open MAIL, "| /var/qmail/bin/qmail-inject" or &die_html("test");
|print MAIL "[your mail]";
|close MAIL;
|
|I suppose you forgot the full path to qmail-inject ...
|
|Fred
|>
|> Thanks for your help.
|>
|>
|
--
Nicolas MONNET, Technical Director, IT-Xchange
http://www.it-xchange.com>
mailto:[EMAIL PROTECTED]>
mailto:[EMAIL PROTECTED]>
Re: open(FH,'|qmail-inject') fails
> Might be a faq, but why would open(FH,'|qmail-inject') fail with
> fatal: read-error from within mod_perl?
Use
open MAIL, "| /var/qmail/bin/qmail-inject" or &die_html("test");
print MAIL "[your mail]";
close MAIL;
I suppose you forgot the full path to qmail-inject ...
Fred
>
> Thanks for your help.
>
>
