qmail Digest 10 Apr 1999 10:00:01 -0000 Issue 606
Topics (messages 24071 through 24110):
modify outgoing headers
24071 by: "Greg Owen" <[EMAIL PROTECTED]>
24097 by: Greg Owen {gowen} <[EMAIL PROTECTED]>
Allow -ext on with qmail-users (virt user?)
24072 by: Harald Hanche-Olsen <[EMAIL PROTECTED]>
qmail speed
24073 by: Dave Sill <[EMAIL PROTECTED]>
24074 by: [EMAIL PROTECTED]
24076 by: Harald Hanche-Olsen <[EMAIL PROTECTED]>
24077 by: Dave Sill <[EMAIL PROTECTED]>
24078 by: [EMAIL PROTECTED]
24079 by: [EMAIL PROTECTED]
24080 by: Russell Nelson <[EMAIL PROTECTED]>
24081 by: "Richard Shetron" <[EMAIL PROTECTED]>
24082 by: [EMAIL PROTECTED]
24083 by: [EMAIL PROTECTED]
24085 by: "Sam" <[EMAIL PROTECTED]>
24086 by: [EMAIL PROTECTED] (Lorens Kockum)
24087 by: [EMAIL PROTECTED]
24088 by: [EMAIL PROTECTED]
24089 by: [EMAIL PROTECTED]
24090 by: David Villeger <[EMAIL PROTECTED]>
24091 by: "Chris Garrigues" <[EMAIL PROTECTED]>
24092 by: [EMAIL PROTECTED]
24093 by: Mark Delany <[EMAIL PROTECTED]>
24094 by: Mark Delany <[EMAIL PROTECTED]>
24096 by: Dave Sill <[EMAIL PROTECTED]>
Autoreply for incoming email.
24075 by: Bruno Wolff III <[EMAIL PROTECTED]>
24084 by: [EMAIL PROTECTED] (Lorens Kockum)
24095 by: Giles Lean <[EMAIL PROTECTED]>
24098 by: "Fred Lindberg" <[EMAIL PROTECTED]>
qmail speed report
24099 by: [EMAIL PROTECTED]
tar pitting
24100 by: [EMAIL PROTECTED]
24102 by: xs <[EMAIL PROTECTED]>
Can't send from other place to qmail
24101 by: Juan Carlos Castro y Castro <[EMAIL PROTECTED]>
dot-qmail and Maildir
24103 by: "Reid Sutherland" <[EMAIL PROTECTED]>
24104 by: Russell Nelson <[EMAIL PROTECTED]>
24105 by: "Reid Sutherland" <[EMAIL PROTECTED]>
24106 by: Harald Hanche-Olsen <[EMAIL PROTECTED]>
Qmail and content filtering?
24107 by: Qmail <[EMAIL PROTECTED]>
24110 by: Lance Ware <[EMAIL PROTECTED]>
[Q] qmail speed "again"
24108 by: Silver CHEN <[EMAIL PROTECTED]>
Qmail IMAP4 Setup Instruction
24109 by: "Postmaster" <[EMAIL PROTECTED]>
Administrivia:
To subscribe to the digest, e-mail:
[EMAIL PROTECTED]
To unsubscribe from the digest, e-mail:
[EMAIL PROTECTED]
To bug my human owner, e-mail:
[EMAIL PROTECTED]
To post to the list, e-mail:
[EMAIL PROTECTED]
----------------------------------------------------------------------
> OK.. cool... except that i don't see how FAQ 5.5 has anything to do > with it. Still, if i can get away without modifying any binaries, i'll > be happy.... > ...[FAQ 5.5] > Third, follow the procedure in question 5.4, but set RELAYCLIENT to the > string ``@fixme'': > > tcp-env: 1.2.3.6, 1.2.3.7: setenv = RELAYCLIENT @fixme > > Here 1.2.3.6 and 1.2.3.7 are the clients' IP addresses. If you are using > tcpserver instead of inetd and tcpd, put > > 1.2.3.6:allow,RELAYCLIENT="@fixme" > 1.2.3.7:allow,RELAYCLIENT="@fixme" > > into /etc/tcp.smtp, and run tcprules as in question 5.4. I think the gist of what he's saying is that you can use the variable setting capabilities of tcpserver (shown here modifying "RELAYCLIENT") to set other variables. One of these variables modifies the Received line's "from [host]", and another modifies the IP address associated with that host. I'm not being vague for the hell of it; I just don't have access to my system which uses this feature right now. If you can wait another 6 hours or so, I'll mail the exact lines that I'm using to do exactly what you're asking for. -- gowen -- Greg Owen -- [EMAIL PROTECTED] -- [EMAIL PROTECTED] Please note my new [EMAIL PROTECTED] address which will become my default address in March, and which works now.
On Fri, 9 Apr 1999, Greg Owen wrote: > I think the gist of what he's saying is that you can use the variable > setting capabilities of tcpserver (shown here modifying "RELAYCLIENT") to > set other variables. One of these variables modifies the Received line's > "from [host]", and another modifies the IP address associated with that > host. Okay, I got onto the machine that I'm doing this with right now. Here's the tcp.smtp (tcpserver) line I'm using to masquerade my internal mail store's IP and name. 13.246.76.26:allow,RELAYCLIENT="",TCPREMOTEHOST="mail.scansoft.com", TCPREMOTEIP="192.168.0.1" 13.246.76.26 is the mail store/server's Real IP. allow,RELAYCLIENT="" says this host is allowed to relay. TCPREMOTEHOST and TCPREMOTEIP affect what qmail puts in the Received: headers. Here is the Received: header that results: Received: from mail.scansoft.com (192.168.0.1) by munin.scansoft.com with SMTP; 9 Apr 1999 18:15:19 -0000 Viola. The Received header no longer gives out information about my internal, firewalled mail store, thus denying that tiny tidbit of information to attackers. I found these variables in one of the man pages, presumably from tcpserver, but haven't looked to make sure. -- gowen -- Greg Owen -- [EMAIL PROTECTED] -- [EMAIL PROTECTED] Please note my new [EMAIL PROTECTED] address which will become my default address in March, and which works now.
+ Sameer Vijay <[EMAIL PROTECTED]>: | [...]Now, instead of having an address of the sort sameer-iitb-*@, I | would like it to have as iitb-*@ without having an account for the | iitb as user. [...] | | I thought of using qmail-users and added the following line | | +iitb-:sameer:501:100:/home/sameer/:-:sameer: This means that mail to iitb-foobar will be handled by ~sameer/.qmail-sameerfoobar which is probably not what you want. Try these instead: =iitb::sameer:501:100:/home/sameer:-:iitb: +iitb-::sameer:501:100:/home/sameer:-:iitb-: which means that mail to iitb is handled by ~sameer/.qmail-iitb and mail to iitb-foobar by ~sameer/.qmail-iitb-foobar. - Harald
[EMAIL PROTECTED] wrote: >On Thu, Apr 08, 1999 at 03:59:18PM -0400, Dave Sill wrote: >> [EMAIL PROTECTED] wrote: >> >> >Current speed is 20,000-40,000/hour on a PPRO200/PII350 >> >> Which is it? PPRO200 or PII350? > >20K/hour for PPRO200 >40K/hour for PII350 Well, you can get 60k/hour using both. Frankly, these look like pretty good numbers to me. >> How's your qmail configured? What does qmail-showctl say? > >concurrencyremote: Remote concurrency is 255. When you're sending messages, how many qmail-remotes are running? >> What kind of connectivity do you have? > >That particular box is connected to a 100Mbit network (located at Exodus) Is that 100Mb local or Internet? >> Multiple servers? RAM disk? qmail 2.0? > >/tmp is on a RAM disk I was thinking in terms of putting the queue on a Quantum Solid State Disk (http://www.quantum.com/products/ssd/). >Is qmail 2.0 out? No, but it promises dramtically improved queue performance. -Dave
On Fri, Apr 09, 1999 at 08:44:06AM -0400, Dave Sill wrote: > [EMAIL PROTECTED] wrote: > >On Thu, Apr 08, 1999 at 03:59:18PM -0400, Dave Sill wrote: > >> [EMAIL PROTECTED] wrote: > >> > >> >Current speed is 20,000-40,000/hour on a PPRO200/PII350 > >> > >> Which is it? PPRO200 or PII350? > > > >20K/hour for PPRO200 > >40K/hour for PII350 > > Well, you can get 60k/hour using both. Frankly, these look like pretty > good numbers to me. I need to find a way of doing >100K/hour. Ideally with one machine. I vaguely seems to recall that the author of qmail was claiming something like 100K/hour performance? > >> How's your qmail configured? What does qmail-showctl say? > > > >concurrencyremote: Remote concurrency is 255. > > When you're sending messages, how many qmail-remotes are running? About 90-130. 255 if I stop delivery for a little while and then restart it. To me that means that the machine can send faster then I can get the messages into the queue. > >> What kind of connectivity do you have? > > > >That particular box is connected to a 100Mbit network (located at Exodus) > > Is that 100Mb local or Internet? 100Mbit Internet. > >> Multiple servers? RAM disk? qmail 2.0? > > > >/tmp is on a RAM disk > > I was thinking in terms of putting the queue on a Quantum Solid State > Disk (http://www.quantum.com/products/ssd/). > > >Is qmail 2.0 out? > > No, but it promises dramtically improved queue performance. Do we have an ETA? Dirk > -Dave
+ [EMAIL PROTECTED]: | > >Is qmail 2.0 out? | > | > No, but it promises dramtically improved queue performance. | | Do we have an ETA? Nope. Dan is too smart for that. Too many software developers have been burned by suggesting a delivery date, then finding their release slip by weeks, months, even years. Qmail 2 will come when it comes. Don't hold your breath or even make plans to use it: Dan has other stuff on his mind, like teaching and suing the US government. There is one bit of information, though: qmail 2 is supposed to use something called zeroseek to handle the queue. Dan has created a mailing list for discussions of zeroseek, but so far he has released nothing that we could discuss. It may well be a long wait. - Harald
[EMAIL PROTECTED] wrote: > >I need to find a way of doing >100K/hour. Ideally with one machine. I >vaguely seems to recall that the author of qmail was claiming something >like 100K/hour performance? I can't find anything like on the web page. The closest claims are: Efficient: On a Pentium under BSD/OS, qmail can easily sustain 200000 local messages per day---that's separate messages injected and delivered to mailboxes in a real test! SPEED---qmail blasts through mailing lists two orders of magnitude faster than sendmail. For example, each message on the qmail mailing list is delivered to more than 1000 hosts around the world in just 76 seconds. Scheduling: I sent a message to 8192 ``trash'' recipients on my home machine. All the deliveries were done in a mere 78 seconds---a rate of over 9 million deliveries a day! >> When you're sending messages, how many qmail-remotes are running? > >About 90-130. 255 if I stop delivery for a little while and then restart >it. To me that means that the machine can send faster then I can get the >messages into the queue. That's what it sounds like. So you either need to speed up the existing process via h/w or s/w changes, or modify the injection process. If you have money to throw at it, a solid state disk for the queue would help a lot. Moving the queue to a tmpfs or removing fsync() calls, as others have suggested, should help. Maybe you should hack up a customized version of qmail-queue that would inject your messages faster--perhaps while qmail is stopped. -Dave
Made a new machine to send out mailings. PII 450 with 1GB of RAM. I will try the "no fsync" way of doing things today. Its Friday and a mailing needs to go out. Preliminary tests show that "no fsync" seems to work so we'll see what the real life throughput improvement will be. Next experiment will be to run multiple independent copies of qmail where each copy has its own queue on a 60MByte RAMdisk. In theory this should result in up to 255*10 concurrent connections to the outside world. If neither gets over 100K deliveries per hour then I'll have to write my own "gattling gun" SMTP delivery thingy and just feed messages to qmail that have problems. Results will be reported back to the list. Dirk On Fri, Apr 09, 1999 at 11:07:31AM -0400, Dave Sill wrote: > [EMAIL PROTECTED] wrote: > > > >I need to find a way of doing >100K/hour. Ideally with one machine. I > >vaguely seems to recall that the author of qmail was claiming something > >like 100K/hour performance? > > I can't find anything like on the web page. The closest claims are: > > Efficient: On a Pentium under BSD/OS, qmail can easily sustain 200000 > local messages per day---that's separate messages injected and > delivered to mailboxes in a real test! > > SPEED---qmail blasts through mailing lists two orders of magnitude > faster than sendmail. For example, each message on the qmail mailing > list is delivered to more than 1000 hosts around the world in just > 76 seconds. > > Scheduling: I sent a message to 8192 ``trash'' recipients on my home > machine. All the deliveries were done in a mere 78 seconds---a rate > of over 9 million deliveries a day! > > >> When you're sending messages, how many qmail-remotes are running? > > > >About 90-130. 255 if I stop delivery for a little while and then restart > >it. To me that means that the machine can send faster then I can get the > >messages into the queue. > > That's what it sounds like. So you either need to speed up the > existing process via h/w or s/w changes, or modify the injection > process. > > If you have money to throw at it, a solid state disk for the queue > would help a lot. > > Moving the queue to a tmpfs or removing fsync() calls, as others have > suggested, should help. > > Maybe you should hack up a customized version of qmail-queue that would > inject your messages faster--perhaps while qmail is stopped. > > -Dave
Got it. That's the one problem when mathematicans start to program. Usually very little attention is being paid to performance optimizations. One can actually see that in qmail. Well thought out and very modular. Lots of performance expensive design decisions though. If SPAMmers can do 200K messages per hour then we should be able to do that as well when sending legit stuff! Dirk PS. Hehe... yes, I'm trying to provoke somebody into more performance.... ;) On Fri, Apr 09, 1999 at 04:42:32PM +0200, Harald Hanche-Olsen wrote: > + [EMAIL PROTECTED]: > > | > >Is qmail 2.0 out? > | > > | > No, but it promises dramtically improved queue performance. > | > | Do we have an ETA? > > Nope. Dan is too smart for that. Too many software developers have > been burned by suggesting a delivery date, then finding their release > slip by weeks, months, even years. Qmail 2 will come when it comes. > Don't hold your breath or even make plans to use it: Dan has other > stuff on his mind, like teaching and suing the US government. > > There is one bit of information, though: qmail 2 is supposed to use > something called zeroseek to handle the queue. Dan has created a > mailing list for discussions of zeroseek, but so far he has released > nothing that we could discuss. It may well be a long wait. > > - Harald
[EMAIL PROTECTED] writes: > That's the one problem when mathematicans start to program. > Usually very little attention is being paid to performance > optimizations. > > One can actually see that in qmail. Well thought out and > very modular. Lots of performance expensive design decisions though. Like reliability? It's a tolerable cost. > If SPAMmers can do 200K messages per hour then we should be > able to do that as well when sending legit stuff! But spammers treat 4xx and 5xx errors equally. There's no point in queuing something since they're going to lose their account in a few hours. > PS. Hehe... yes, I'm trying to provoke somebody into more > performance.... ;) Through insults? Not bloody likely. -- -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson Crynwr supports Open Source(tm) Software| PGPok | There is good evidence 521 Pleasant Valley Rd. | +1 315 268 1925 voice | that freedom is the Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | cause of world peace.
Things that may improve things is to look at caching/high speed scsi controllers, such as the DPT 3334UW and the icp-vortex controllers. The DPT takes up to 64MB/controller and the icp-vortex up to 128MB/controller. We're using 2 DPT 3334UW's on our news machine (40GB spool) at this time and it's handling a T1 worth of feed without problems on 128MB RAM and an AMD K6-450 (usually runs 95% idle). It only slow down when doing an expire. > Made a new machine to send out mailings. PII 450 with 1GB of RAM. > > I will try the "no fsync" way of doing things today. Its Friday and > a mailing needs to go out. Preliminary tests show that "no fsync" > seems to work so we'll see what the real life throughput improvement > will be. > > Next experiment will be to run multiple independent copies of qmail > where each copy has its own queue on a 60MByte RAMdisk. In theory > this should result in up to 255*10 concurrent connections to the > outside world. > > If neither gets over 100K deliveries per hour then I'll have to > write my own "gattling gun" SMTP delivery thingy and just feed > messages to qmail that have problems. > > Results will be reported back to the list. > > Dirk > > > On Fri, Apr 09, 1999 at 11:07:31AM -0400, Dave Sill wrote: > > [EMAIL PROTECTED] wrote: > > > > > >I need to find a way of doing >100K/hour. Ideally with one machine. I > > >vaguely seems to recall that the author of qmail was claiming something > > >like 100K/hour performance? > > > > I can't find anything like on the web page. The closest claims are: > > > > Efficient: On a Pentium under BSD/OS, qmail can easily sustain 200000 > > local messages per day---that's separate messages injected and > > delivered to mailboxes in a real test! > > > > SPEED---qmail blasts through mailing lists two orders of magnitude > > faster than sendmail. For example, each message on the qmail mailing > > list is delivered to more than 1000 hosts around the world in just > > 76 seconds. > > > > Scheduling: I sent a message to 8192 ``trash'' recipients on my home > > machine. All the deliveries were done in a mere 78 seconds---a rate > > of over 9 million deliveries a day! > > > > >> When you're sending messages, how many qmail-remotes are running? > > > > > >About 90-130. 255 if I stop delivery for a little while and then restart > > >it. To me that means that the machine can send faster then I can get the > > >messages into the queue. > > > > That's what it sounds like. So you either need to speed up the > > existing process via h/w or s/w changes, or modify the injection > > process. > > > > If you have money to throw at it, a solid state disk for the queue > > would help a lot. > > > > Moving the queue to a tmpfs or removing fsync() calls, as others have > > suggested, should help. > > > > Maybe you should hack up a customized version of qmail-queue that would > > inject your messages faster--perhaps while qmail is stopped. > > > > -Dave > -- Richard Shetron [EMAIL PROTECTED] [EMAIL PROTECTED] What is the Meaning of Life? There is no meaning, It's just a consequence of complex carbon based chemistry; don't worry about it The Super 76, "Free Aspirin and Tender Sympathy", Las Vegas Strip.
[EMAIL PROTECTED] writes: > That's the one problem when mathematicans start to program. > Usually very little attention is being paid to performance > optimizations. Cough. I take exception to that! Though I'm no DJB, the PhD after my name means I can claim to be a mathematician. The performance of qmail is quite impressive--and a major reason for qmail 2 is to improve it even more. Suggesting that DJB doesn't pay attention to performance is just stupid. > One can actually see that in qmail. Well thought out and > very modular. Lots of performance expensive design decisions though. Like what? Do I smell the fork/exec flame war smoldering afresh? > If SPAMmers can do 200K messages per hour then we should be able to > do that as well when sending legit stuff! You can. Just make sure that you send a single email, with a CC list of size 200K. Then you can get performance comparable to spammers. Len Budney, PhD -- I wasn't talking about sendmail+shell versus sendmail. I said you would need dozens of subshells to make _qmail_ as slow as sendmail. -- Prof. Dan Bernstein
[EMAIL PROTECTED] writes: > If neither gets over 100K deliveries per hour then I'll have to > write my own "gattling gun" SMTP delivery thingy and just feed > messages to qmail that have problems. Are you sure you're not a spammer? You're performance needs seem to be mind-boggling, yet you refuse to throw more hardware at the problem (which I would expect any large, legitimate business to do). -- 103. In Company of your Betters be not longer in eating than they are lay not your Arm but only your hand upon the table. -- George Washington, "Rules of Civility & Decent Behaviour"
Dave Sill writes: > >> When you're sending messages, how many qmail-remotes are running? > > > >About 90-130. 255 if I stop delivery for a little while and then restart > >it. To me that means that the machine can send faster then I can get the > >messages into the queue. > > That's what it sounds like. So you either need to speed up the > existing process via h/w or s/w changes, or modify the injection > process. I find this very difficult to believe, that it takes longer to add something to the queue than to make several DNS queries, establish a socket, and go through the SMTP handshake in order to deliver the message. Either you're running out of some resource - memory or sockets - which throttles your output, or your injection mechanism is piss-poor inefficient. If you're injecting messages one at a time, it is possible that fsyncs and the rest of the disk activity are throttling your input, but I still find it somewhat difficult to swallow. However that's the only possible answer in absence of a resource shortage. You should be able to remedy that by injecting multiple messages in parallel. -- Sam
On the qmail list [EMAIL PROTECTED] wrote: > >You can. Just make sure that you send a single email, with a CC list >of size 200K. Then you can get performance comparable to spammers. Just for the sake of discussion, what would be the best way? I'm envisioning using xargs to distribute the rcpt addresses to qmail-inject, which would nicely create one message-instance for every n recipients, but I haven't quite figured out how to get the mail itself on the stdin of qmail-inject without an intermediary. Not sure it's possible, I'd have to look at xargs sources to see if it is, and if not I could patch in a -f option... Not that an intermediary would be a problem, of course, it's just ugly. -- #include <std_disclaim.h> Lorens Kockum
Why is an observation an insult? Its just a pattern that I have seen a few times. No big deal. Anyways, my main concern is to get more speed out of this thing. Computer hardware has reached quite amazing performance levels. It is time that software catches up. Dirk On Fri, Apr 09, 1999 at 03:59:39PM -0000, Russell Nelson wrote: > [EMAIL PROTECTED] writes: > > That's the one problem when mathematicans start to program. > > Usually very little attention is being paid to performance > > optimizations. > > > > One can actually see that in qmail. Well thought out and > > very modular. Lots of performance expensive design decisions though. > > Like reliability? It's a tolerable cost. > > > If SPAMmers can do 200K messages per hour then we should be > > able to do that as well when sending legit stuff! > > But spammers treat 4xx and 5xx errors equally. There's no point in > queuing something since they're going to lose their account in a few > hours. > > > PS. Hehe... yes, I'm trying to provoke somebody into more > > performance.... ;) > > Through insults? Not bloody likely. > > -- > -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson > Crynwr supports Open Source(tm) Software| PGPok | There is good evidence > 521 Pleasant Valley Rd. | +1 315 268 1925 voice | that freedom is the > Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | cause of world peace.
Yes, very sure that I'm not a SPAMmer. Neither is my customer. Some people just like to get information on certain subjects and sign up for these newsletters. Problem is that the information is time sensitive so everybody needs to get it at about the same time. At least on the same day. Of course the problem could be solved by having 20 boxes do it in parallel. But where is the fun in that. Also rack space goes for $1000/rack/month where these boxes live. No need to waste money just because they could. After all, finding cost effective solutions is what they pay us for. Dirk On Fri, Apr 09, 1999 at 12:12:09PM -0400, [EMAIL PROTECTED] wrote: > [EMAIL PROTECTED] writes: > > > If neither gets over 100K deliveries per hour then I'll have to > > write my own "gattling gun" SMTP delivery thingy and just feed > > messages to qmail that have problems. > > Are you sure you're not a spammer? You're performance needs seem to be > mind-boggling, yet you refuse to throw more hardware at the problem > (which I would expect any large, legitimate business to do). > > -- > 103. In Company of your Betters be not longer in eating than they are > lay not your Arm but only your hand upon the table. > -- George Washington, "Rules of Civility & Decent Behaviour"
qmail-inject is too slow. I've been using qmail-queue instead. Dirk On Fri, Apr 09, 1999 at 05:01:48PM -0000, Lorens Kockum wrote: > On the qmail list [EMAIL PROTECTED] wrote: > > > >You can. Just make sure that you send a single email, with a CC list > >of size 200K. Then you can get performance comparable to spammers. > > Just for the sake of discussion, what would be the best way? > > I'm envisioning using xargs to distribute the rcpt addresses to > qmail-inject, which would nicely create one message-instance > for every n recipients, but I haven't quite figured out how > to get the mail itself on the stdin of qmail-inject without > an intermediary. Not sure it's possible, I'd have to look at > xargs sources to see if it is, and if not I could patch in a > -f option... Not that an intermediary would be a problem, of > course, it's just ugly. > > -- > #include <std_disclaim.h> Lorens Kockum
Hi Dirk, I tried the no_fsync method last night and recorded a 10% speed improvement. At 07:27 AM 4/9/99 -0700, [EMAIL PROTECTED] wrote: >On Fri, Apr 09, 1999 at 08:44:06AM -0400, Dave Sill wrote: >> When you're sending messages, how many qmail-remotes are running? > >About 90-130. 255 if I stop delivery for a little while and then restart >it. To me that means that the machine can send faster then I can get the >messages into the queue. In my experience (also sending huge amounts of unique emails) the bottleneck in qmail (for this application) is the queuing system. Look at your todo dir and I bet you'll see thousands of entries there in a single directory (R. Nelson has a patch to split this directory, but it still didn't do what I needed). As long as some messages are waiting to be "preprocessed" (see INTERNALS for explanation), qmail does not achieve 255 simultaneous qmail-remote. Besides, as you inject the messages into the queue, qmail-send spends a huge amount of time cleaning up (via qmail-clean). It seems to me that qmail-send gets too busy preprocessing and cleaning up and has no time to send the emails as fast as they come. Obviously, djb didn't think qmail would be used this way. If he had, why would he have left the todo and intd directories flat? Also, from THOUGHTS: >qmail-send doesn't have any notions of precedence, priority, fairness, >importance, etc. It handles the queue in first-seen-first-served order. >One could put a lot of work into doing something different, but that >work would be a waste: given the triggering mechanism and qmail's >deferral strategy, it is exceedingly rare for the queue to contain more >than one deliverable message at any given moment. The triggering mechanism is the select() call on lock/trigger. That works well, but I still have 10000 messages waiting to be preprocessed! Everybody says: > buy this or that, faster disks, RAM disks, scsi, .... Of course buying more, better hardware will improve performance, but I still think an architectural change on the way qmail-send preprocesses and then sends emails would be better (why not 2 processes: qmail-preprocessor and qmail-send?). Spending more money when other solutions can be made available is not a good solution. What I'd like is the opposite: *saving* money while improving (or preserving) performance! I'd love to hear suggestions. >> >Is qmail 2.0 out? >> >> No, but it promises dramtically improved queue performance. Yes, zeroseek technology. We are all impatiently waiting for it. David.
> From: [EMAIL PROTECTED] > Date: Fri, 9 Apr 1999 10:13:43 -0700 > > Of course the problem could be solved by having 20 boxes do it in > parallel. But where is the fun in that. Also rack space goes for > $1000/rack/month where these boxes live. No need to waste money just > because they could. After all, finding cost effective solutions is what > they pay us for. Get rackmounted netwinders. Two of them in 2 inches of rackspace. Chris -- Chris Garrigues virCIO +1 512 432 4046 4314 Avenue C O- http://www.DeepEddy.Com/~cwg/ Austin, TX 78751-3709 +1 512 374 0500 My email address is an experiment in SPAM elimination. For an explanation of what we're doing, see http://www.DeepEddy.Com/tms.html Nobody ever got fired for buying Microsoft, but they could get fired for relying on Microsoft.
PGP signature
[EMAIL PROTECTED] <[EMAIL PROTECTED]> writes on 9 April 1999 at 08:53:51 -0700 > Got it. > > That's the one problem when mathematicans start to program. > Usually very little attention is being paid to performance > optimizations. > > One can actually see that in qmail. Well thought out and > very modular. Lots of performance expensive design decisions though. Reliability is a goal taking priority over speed for qmail, which may explain the design decisions you're seeing. > If SPAMmers can do 200K messages per hour then we should be > able to do that as well when sending legit stuff! Spammers don't care about reliability; or rather, they'd be happy with 98% or 99% and a high rate of sending. I, on the other hand, would NOT find 99% success at delivering mail satisfactory. > PS. Hehe... yes, I'm trying to provoke somebody into more > performance.... ;) Dan is supposedly hard at work on qmail 2.0, using "zeroseek" queueing to produce greatly improved performance without sacrifice of reliability. Since qmail performance is already at the top of the heap (somewhere around it), a great improvement will be really spectacular! -- David Dyer-Bennet [EMAIL PROTECTED] http://www.ddb.com/~ddb (photos, sf) Minicon: http://www.mnstf.org/minicon http://ouroboros.demesne.com/ The Ouroboros Bookworms Join the 20th century before it's too late!
At 10:19 AM Friday 4/9/99, [EMAIL PROTECTED] wrote: >qmail-inject is too slow. I've been using qmail-queue instead. Only because you're doing an invocation per recipient! I'm guessing, but I'm almost certain the qmail-qstat will show some interesting numbers. Particularly "messages in queue but not yet preprocessed" value. What's particularly killing your system is the message injection process. >On Fri, Apr 09, 1999 at 05:01:48PM -0000, Lorens Kockum wrote: >> On the qmail list [EMAIL PROTECTED] wrote: >> > >> >You can. Just make sure that you send a single email, with a CC list >> >of size 200K. Then you can get performance comparable to spammers. >> >> Just for the sake of discussion, what would be the best way? Use qmail-inject with multiple Bcc: recipients as suggested a few days ago. Since the invocation happens just the once for the all recipients, there is no advantage to using qmail-queue (and some disadvantages if you ask me). On the matter of comparison to spammers, here's what you need to do to get comparable results: 1. Turn off all disk I/O 2. Ignore the SMTP transaction 3. Don't care if a recipient sees the mail zero or more times 4. Ignore system and network failures If you want to go part way down this path I suggest putting /var/qmail/queue on a memory-based file system rather than twiddling fsync() calls in the code. FWIW. The best I've seen out of a single box Pentium with one or two high speed spindles is around 100K per hour. The systems tend to run out of queue disk I/O. (This of course is gross generalization as most people will realise, but it gives a ballpark expectation for an unmodified qmail system). >> I'm envisioning using xargs to distribute the rcpt addresses to No point. Put the recipients in bcc: headers and only invoke qmail-inject the once. >> qmail-inject, which would nicely create one message-instance >> for every n recipients, but I haven't quite figured out how >> to get the mail itself on the stdin of qmail-inject without See the example I posted the other day. Regards.
At 04:27 PM Friday 4/9/99, Sam wrote: >Dave Sill writes: > >> >> When you're sending messages, how many qmail-remotes are running? >> > >> >About 90-130. 255 if I stop delivery for a little while and then restart >> >it. To me that means that the machine can send faster then I can get the >> >messages into the queue. >> >> That's what it sounds like. So you either need to speed up the >> existing process via h/w or s/w changes, or modify the injection >> process. > >I find this very difficult to believe, that it takes longer to add >something to the queue than to make several DNS queries, establish a >socket, and go through the SMTP handshake in order to deliver the message. I don't find it particularly hard to believe. queue-insertion is expension on the file system and slow if the file system is busy with 100s of outbound deliveries (of individual mails). At, say, 200K per hour, that's 50 queue insertions per second and 50 queue removals per second. That's starting to be a lot of I/O. >If you're injecting messages one at a time, it is possible that fsyncs and >the rest of the disk activity are throttling your input, but I still find >it somewhat difficult to swallow. However that's the only possible answer >in absence of a resource shortage. You should be able to remedy that by >injecting multiple messages in parallel. I doubt it. He's probably out of disk thruput. Of course presenting iostats and vmstats whilst the processes are running would clear a lot of that up. Regards.
David Villeger <[EMAIL PROTECTED]> wrote: > >As long as some messages are waiting to be "preprocessed" (see INTERNALS >for explanation), qmail does not achieve 255 simultaneous qmail-remote. >Besides, as you inject the messages into the queue, qmail-send spends a >huge amount of time cleaning up (via qmail-clean). It seems to me that >qmail-send gets too busy preprocessing and cleaning up and has no time to >send the emails as fast as they come. The real (reliable, general-purpose) fix for this will be 2.0/zeroseek, I think. A short-term, special purpose fix for "customized spam" applications would be a high-speed queuer that does the queuing and preprocessing of qmail-queue and qmail-send. E.g., 1) stop qmail 2) invoke custom queuer which: a) puts cutomized message in queue/mess/NNN/MMM b) puts envelope sender in queue/info/NNN/MMM c) puts recipient in queue/remote/NNN/MMM 3) restart qmail Cuts out all the preprocessing junk: rewriting addresses, deciding if local or remote, todo files, etc. Avoids all the work necessary to keep the queue sane during the injection process. >Obviously, djb didn't think qmail would be used this way. If he had, why >would he have left the todo and intd directories flat? My batch queuer design avoids both intd and todo altogether. But, yes, qmail was clearly not designed for maximal injection rates. -Dave
It is very important that autoresponders not reply to certain addresses, such as <>. Also for most applications they should be rate limited to something like one message a day. If the address is potentially on a mailing list (as is the case for most people's mailboxes), then the program should also try to determine which messages are list messages and not reply to them. You can do this by checking the recipient headers for the user's address(es). Checking the precedence header for bulk or junk. Checking for an envelope sender with a local part ending in -owner. It would be nice if there was an rfc for autoresponders out there, as there seem to be more and more people using braindead ones.
On the qmail list [EMAIL PROTECTED] wrote: >If the address is potentially on a mailing list (as is the case for most >people's mailboxes), then the program should also try to determine which >messages are list messages and not reply to them. You can do this by checking >the recipient headers for the user's address(es). Checking the precedence >header for bulk or junk. Checking for an envelope sender with a local part >ending in -owner. Your list seems to cover most bases: man vacation: % No message is sent if the `To:' or the `Cc:' line does not % list the user to whom the original message was sent or one % of a number of aliases for them, if the initial From line % includes the string -REQUEST@, or if a `Precedence: bulk' or % `Precedence: junk' line is included in the header. % % Sun Release 4.1 Last change: 25 November 1987 Seems the BSD folks thought more was necessary, seems reasonable too: % No message will be sent unless login (or an alias supplied % using the -a option) is part of either the ``To:'' or ``Cc:'' % headers of the mail. No messages from ``???-REQUEST'', % ``Postmaster'', ``UUCP'', ``MAILER'', or ``MAILER-DAEMON'' % will be replied to (where these strings are case insensitive) % nor is a notification sent if a ``Precedence: bulk'' or % ``Precedence: junk'' line is included in the mail headers. % % 4.3 Berkeley Distribution June 16, 1993 *-request seems more reasonable than *-owner, but why not both. Be conservative in what you send and liberal in what you receive => in this case put -request on mailing lists, but don't send acks to -owner. Or do, now that I think of it, owner is supposed to be a human and request is supposed to be a machine... so vacation should be good. Mailing lists using VERP could not rely on that without hacking of course, but bulk would be sufficient (why not "list", seems I've seen that a lot, maybe not recently though, is there a list of things to put in Precedence: other than bulk and junk?). >It would be nice if there was an rfc for autoresponders out there, as there >seem to be more and more people using braindead ones. But there is one that is applicable... % RFC 1123, section 5.3.3, Reliable Mail Receipt % % If there is a delivery failure after acceptance of a message, % the receiver-SMTP MUST formulate and mail a notification % message. This notification MUST be sent using a null ("<>") % reverse path in the envelope; see Section 3.6 of RFC-821. The % recipient of this notification SHOULD be the address from the % envelope return path (or the Return-Path: line). However, if % this address is null ("<>"), the receiver-SMTP MUST NOT send a % notification. If the address is an explicit source route, it % SHOULD be stripped down to its final hop. I know that only refers to delivery *failures*, but a vacation program is in essence a notification of temporary delivery failure, isn't it? Following sections in RFC 1123, section 5.3.6, subsection b, and section 5.3.7, subsection E, specifically address and explain the reasoning behind this SHOULD rule, based on mailing lists. I am sorry it is a SHOULD, but I suppose the writer had to take into account old systems with no proviso for Return-Paths. So, to begin with no vacation notification is sent if the message is from a mailing-list, based on the above criteria; second, if some other kind of notification must be sent (mailbox does not exist) it should go to the envelope return address anyway, which is the mailing list administrator. Too many pieces of shit do not observe the extremely simple rules laid down in the above paragraph. -- #include <std_disclaim.h> Lorens Kockum
On Fri, 9 Apr 1999 09:31:50 -0500 Bruno Wolff III wrote: > It is very important that autoresponders not reply to certain addresses, > such as <>. Also for most applications they should be rate limited to > something like one message a day. > ... > It would be nice if there was an rfc for autoresponders out there, as there > seem to be more and more people using braindead ones. See http://www.pobox.com/~djb/proto/mailloops.txt. There is a qmail specific rate limited autoresponder linked on the qmail.org web page: http://www.netmeridian.com/e-huss/autorespond.tar.gz I have written one -- also rate limited -- as I needed one I could use with "that other mailer": http://www.nemeton.com.au/sw/autoreply/ (Known problem with 1.0beta: create $HOME/.autoreply before you use it. The release version will create this directory if required.) Note I don't think either program will stand up well to thousands of items of mail per hour as currently implemented due to unsophisticated handling of the history records. For lower volume should be fine, or if not, yell. :-) Regards, Giles
On Fri, 9 Apr 1999 09:31:50 -0500, Bruno Wolff III wrote: >It would be nice if there was an rfc for autoresponders out there, as there >seem to be more and more people using braindead ones. It would be nice if all autoresponders added a standard header to outgoing mail. Precedence: junk is used by some, but for others there is no way to know that the message is from a robot. -Sincerely, Fred (Frederik Lindberg, Infectious Diseases, WashU, St. Louis, MO, USA)
FYI Removing the fsync calls results in a speedup of 25-40% for us. Am now getting about 50K/hour on the PPRO200, 128MB RAM and 9GB UW HD. No bad but still quite a ways to go. Next test will be multiple independent mail-queue's on RAM disks. Dirk PS: The faster system (PII 450) is not deployed yet.
Somebody brought up the thought that tarpitting on the receiving end might slow things down. Are there any stats on how many people use tarpitting on their servers? Dirk
i use tarpitting here i limit incoming and outgoing to 100 rcpts per someone saying it was in the rfc, which i have yet to find, ideas? anyway, i figure if a user wants to send to more than 100 rcpts they need a mailing list. -xs end +-------------------------------------+ |Greg Albrecht KF4MKT [EMAIL PROTECTED]| |Safari Internet Fort Lauderdale, FL| |www.safari.net 888-537-9550| +------L-O-W-E-R--D-O-T--O-R-G--------+ On Fri, 9 Apr 1999 [EMAIL PROTECTED] wrote: >Somebody brought up the thought that tarpitting on the receiving end >might slow things down. Are there any stats on how many people use >tarpitting on their servers? > >Dirk >
We installed an intranet into a residence building, the server uses qmail. When some guy from a company sends mail to an account in our building, it bounces back like this. What the @#$%& is happening? ----- Transcript of session follows ----- 421 csp.org.br (red)... Deferred: Not owner 554 <[EMAIL PROTECTED]>... 550 Host unknown (Authoritative answer from name server): Not owner ----- Unsent message follows ----- Received: by cerbero.petrobras.com.br; (5.65v4.0/1.3/10May95) id AA27272; Fri, 9 Apr 1999 16:48:58 -0300 Received: from w53bx77.cenpes.petrobras.com.br (164.85.53.93) by correio.cenpes.petrobras.com.br with SMTP (Eudora Internet Mail Server 1.2); Fri, 9 Apr 1999 15:46:39 -0300 Message-Id: <3.0.1.32.19990409154256.0069772c@correio> X-Sender: ayabe@correio X-Mailer: Windows Eudora Light Version 3.0.1 (32) Date: Fri, 09 Apr 1999 15:42:56 -0300 To: [EMAIL PROTECTED] From: Celso Ayabe <[EMAIL PROTECTED]> Subject: teste do novo e-mail Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Oi Cissita Estou enviando para testar o novo e-mail. Beijos Papito
In my ~alias/.qmail-staff I have it pointing to /home/reid/Maildir/ and qmail won't dump the mail in there. Do I have to use .qmail-local-staff? /home/reid/Maildir is chmod 700 and owned by user reid Thanks. Reid Sutherland Network Administrator ISYS Technology Inc. http://www.isys.ca Fingerprint: 1683 001F A573 B6DF A074 0C96 DBE0 A070 28BE EEA5
Reid Sutherland writes: > In my ~alias/.qmail-staff I have it pointing to /home/reid/Maildir/ and > qmail won't dump the mail in there. Do I have to use .qmail-local-staff? > > /home/reid/Maildir is chmod 700 and owned by user reid What does the log file say? -- -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson Crynwr supports Open Source(tm) Software| PGPok | There is good evidence 521 Pleasant Valley Rd. | +1 315 268 1925 voice | that freedom is the Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | cause of world peace.
Unable to chdir to Maildir Reid Sutherland Network Administrator ISYS Technology Inc. http://www.isys.ca Fingerprint: 1683 001F A573 B6DF A074 0C96 DBE0 A070 28BE EEA5 -----Original Message----- From: [EMAIL PROTECTED] <[EMAIL PROTECTED]> To: Reid Sutherland <[EMAIL PROTECTED]> Date: Friday, April 09, 1999 5:28 PM Subject: Re: dot-qmail and Maildir >On Fri, Apr 09, 1999 at 04:26:51PM -0400, Reid Sutherland wrote: >> In my ~alias/.qmail-staff I have it pointing to /home/reid/Maildir/ and >> qmail won't dump the mail in there. Do I have to use .qmail-local-staff? >> >> /home/reid/Maildir is chmod 700 and owned by user reid > >What does the log say? > >-- >John White johnjohn > at > triceratops.com >PGP Public Key: http://www.triceratops.com/john/public-key.pgp
+ "Reid Sutherland" <[EMAIL PROTECTED]>: | In my ~alias/.qmail-staff I have it pointing to /home/reid/Maildir/ | and qmail won't dump the mail in there. Do I have to use | .qmail-local-staff? | | /home/reid/Maildir is chmod 700 and owned by user reid + Russell Nelson <[EMAIL PROTECTED]>: | What does the log file say? + "Reid Sutherland" <[EMAIL PROTECTED]>: | Unable to chdir to Maildir Yep, that qmail-local is run by the alias user, so user alias must have write access to the maildir, which is not the case since it is mode 700 and owned by you. Even if you make the maildir and its subdirs writable by the alias user, you lose because the files therein will be owned by the alias user with mode 600, so you won't be able to read them without some work. Better to let the message pass through qmail once more: Let ~alias/.qmail-staff forward the mail to reid-staff and do echo './Maildir/' > ~/.qmail-staff as user reid. - Harald
Hi Folks, I'm looking for the best place to insert a content filter into the qmail process for inbound mail. I need to filter inbound SMTP for only certain domains by routing them through an application we wrote. I then need to be able to deliver the message locally or remotely. It looks like in between qmail-queue and qmail-send is the best place. Opinions? Lance Ware WareNet http://www.ware.net/ 949-367-9862 949-348-8668 fax
Is this the correct place to do ot if I want to keep all mail, just insert some text to the body on certain triggers? Lance -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Friday, April 09, 1999 8:53 PM To: Qmail Subject: Re: Qmail and content filtering? On Fri, Apr 09, 1999 at 06:57:32PM -0700, Qmail wrote: > I'm looking for the best place to insert a content filter into the qmail > process for inbound mail. > > I need to filter inbound SMTP for only certain domains by routing them > through an application we wrote. I then need to be able to deliver the > message locally or remotely. > > It looks like in between qmail-queue and qmail-send is the best place. Nope. Between qmail-smtpd and qmail-queue. -- John White johnjohn at triceratops.com PGP Public Key: http://www.triceratops.com/john/public-key.pgp
Dear Sir: * I'm here not for another blamming-war, just some experience. Yes, I'm a very big 'newsletter' system administrator. Now I have to send 400K+ newsletter in less than 8 hours (not a day, since some news are time-intensive) per day. In fact, I'm a new comer for qmail, and our system is now on one single ultra2 workstation w/ 256MB RAM and 2*2GB U-SCSI disks. This is a very old model, about 2 years ago, so the computing power is surely weak than today's PII/PIII, even some P-200+ I think. Can I achieve my target? Yes, I can send 400K+ letters in less than 8 hours. But unfortunately, it works on sendmail 8.9.x, not qmail. The mail reason that I can't switch to qmail is that I'm NOT familiar with qmail in early days, so I chose sendmail. But now I 'almost' give up sendmail, because I can't add/change functions on it via a simple way - I mean something like qmail. I have the bat-book about sendmail, but I only use it to tune the 'options'. I've read all the articles about the topic 'qmail speed'. Now I'm wonder that if qmail can do this too? If the design spirit of qmail can't afford such high-loading task in short time, then sendmail takes the chance back. I'll try to move my system to qmail recently, and give it a try, I think now I have more control on qmail than before, does 40K+ letters in 8- hour possible? I might have some comments soon. My new machine is a PII-350, w/ 256MB RAM and 2*4G UW-SCSI disks, the OS is FreeBSD 3.1. As I know, hotmail use qmail as its base(?) but hotmail is slow, slow, and unreliable 'sometimes'. I don't know what will happen if hotmail choose sendmail as its base now, but I'm curious on this topic. Sorry for this posting if not suitable here, and thanks for any kind of comments about this. Regards. -- Silver CHEN 1999/4/10
I am trying to find documentation on setting IMAP4. So far all my resources are bleed dry. Can someone point me to where to go for this information.
Thanks,
Michael
