Speed up injecting
Hi, my (basic) installation of qmail works fine. My problem is to speed up the injection of mails. The system I am setting up must send mail to different people based on external triggers, so in peak times it must send up to 1 different mails to different recipients as soon as possible. Unfortunately, I have two constraints: 1) Solaris (Slowlaris); 2) Only oneprocess caninjects the mails, so there is no concurrency on the injection phase. I've tried to use qmail-queue directly, but I gained very little. It seems that the bottleneck is in the overhead of starting the qmail-inject (or qmail-queue) for every mail, since qmail-send is extermely fast in consuming the queue. What can I do ? Gianni CampanileWeb Bridges S.r.lVia di Priscilla 10100199 Roma - ItalyTel. +39 06 86510 4744Fax +39 06 86510 4943e-mail: [EMAIL PROTECTED]
Re: Speed up injecting
On Wed, Jul 04, 2001 at 11:29:58AM +0200, Gianni Campanile wrote: [snip] 1) Solaris (Slowlaris); 2) Only one process can injects the mails, so there is no concurrency on the injection phase. I've tried to use qmail-queue directly, but I gained very little. It seems that the bottleneck is in the overhead of starting the qmail-inject (or qmail-queue) for every mail, since qmail-send is extermely fast in consuming the queue. What can I do ? For 1): use a real OS. For 2): divide your list in 10 parts of 1000 emails (for example) and handle these normally, but all at the same time. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Speed up injecting
For 1): use a real OS. No chance, it's not my choice and it is a constraint. For 2): divide your list in 10 parts of 1000 emails (for example) and handle these normally, but all at the same time. As I said there is only one process that builds up the mails, and I don't really like to build up batch files or to send all injects in background. I forgot to say that I only need to send mails, not to receive them. I don't think this will make any difference though. Thanks for help - Original Message - From: Peter van Dijk [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, July 04, 2001 11:34 AM Subject: Re: Speed up injecting On Wed, Jul 04, 2001 at 11:29:58AM +0200, Gianni Campanile wrote: [snip] 1) Solaris (Slowlaris); 2) Only one process can injects the mails, so there is no concurrency on the injection phase. I've tried to use qmail-queue directly, but I gained very little. It seems that the bottleneck is in the overhead of starting the qmail-inject (or qmail-queue) for every mail, since qmail-send is extermely fast in consuming the queue. What can I do ? For 1): use a real OS. For 2): divide your list in 10 parts of 1000 emails (for example) and handle these normally, but all at the same time. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Speed up injecting
On Wed, Jul 04, 2001 at 11:53:02AM +0200, Gianni Campanile wrote: For 1): use a real OS. No chance, it's not my choice and it is a constraint. For 2): divide your list in 10 parts of 1000 emails (for example) and handle these normally, but all at the same time. As I said there is only one process that builds up the mails, and I don't really like to build up batch files or to send all injects in background. You want a solution, but you don't like changing anything in your setup. I give up. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Speed up injecting
Gianni Campanile [EMAIL PROTECTED] wrote: You want a solution, but you don't like changing anything in your setup. I can't change OS. Yes, you can. You simply choose not to, or choose not to bother making the case to your superior. You asked for speed, and Felix pointed out one of the big things slowing you down. I'll keep in mind your idea of splitting mails, but my question is: is there a way to prepare a list of all (different) mails and (different) recipients and feed some program with it ? Sure, you could write this yourself. However, it's probably just as easy to fork a dozen (or a hundred) processes which each call qmail-queue (or qmail-inject, or whatever) multiple times. If, for example, qmail-queue would take many messages from stdin (or from a file), it would be fine, since it already takes multiple recipients. qmail-queue takes multiple recipients for a single message; extending it to handle multiple messages would be complicated and pointless. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ ---
Re: Speed up injecting
On Wed, Jul 04, 2001 at 11:29:58AM +0200, Gianni Campanile wrote: Unfortunately, I have two constraints: 1) Solaris (Slowlaris); uh-oh. 2) Only one process can injects the mails, so there is no concurrency on the injection phase. I've tried to use qmail-queue directly, but I gained very little. It seems that the bottleneck is in the overhead of starting the qmail-inject (or qmail-queue) for every mail, since qmail-send is extermely fast in consuming the queue. What can I do ? Someone posted a perl module called qspam or so calling qmail-remote directly and only injecting into the queue if this fails. This should speed up things dramatically. -- * Henning Brauer, [EMAIL PROTECTED], http://www.bsws.de * * Roedingsmarkt 14, 20459 Hamburg, Germany * Unix is very simple, but it takes a genius to understand the simplicity. (Dennis Ritchie)
Re: Speed up injecting
On Wed, Jul 04, 2001 at 05:16:41PM +0200, Henning Brauer wrote: [snip] Someone posted a perl module called qspam or so calling qmail-remote directly and only injecting into the queue if this fails. This should speed up things dramatically. Only when done massively parallel, ofcourse. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Speed up injecting
On Wed, Jul 04, 2001 at 08:17:36AM -0600, Charles Cazabon wrote: [snip] qmail-queue takes multiple recipients for a single message; extending it to handle multiple messages would be complicated and pointless. And would *only* benefit Solaris users who are too lazy to migrate to something that works. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Speed up injecting
Gianni Campanile [EMAIL PROTECTED] writes:
[...]
My problem is to speed up the injection of mails.
The system I am setting up must send mail to different people
based on external triggers, so in peak times it must send
up to 1 different mails to different recipients as soon as possible.
Unfortunately, I have two constraints:
1) Solaris (Slowlaris);
We run qmail just find on Solaris...
2) Only one process can injects the mails, so there is
no concurrency on the injection phase.
It should be not-too-hard to make this one process fork off and run a
fixed number of qmail-inject processes simultaneously. Something to
the tune of:
#define MAX_FORKS 10
int process_count=0;
int kidstat;
while (more_messages())
{
if (process_count = MAX_FORKS)
{
wait(kidstat); /* Do some error checking, too. */
process_count--;
}
switch(fork())
{
case -1: /* Didn't work. */
/* Do some error checking, maybe try to recover */
exit(1);
case 0: /* Child */
send_message(); /* This would call qmail-inject */
exit(0);
default: /* Parent */
process_count++;
break;
}
}
while (process_count)
{
wait(kidstat);
process_count--;
}
This probably isn't exactly right, but should get you started.
If you can make your program use xargs to start up processes, GNU
xargs supports a command-line flag telling it how many processes to
run in parallel.
If all else fails, and the problem really is the overhead from
fork()/exec(), you could probably hack up qmail-inject or qmail-queue
to handle multiple messages at a time, but it looks like you'd be on
your own. Still, shouldn't be horrendously difficult.
Good luck,
ScottG.
