We're in a similar situation at the moment. However, we want to send out
100,000 UNIQUE emails per day, expanding to 500,000 or more in the near
future. Also, our send window is only actually a couple of hours.
I'm trying to work out the best settings for the concurrencyremote and
conf-split parameters. Our system is a HP Netserver 2000r PIII-667 RAID5
running Linux. Are there any problems in setting conf-split to a very large
value? Is it necessary on a Linux system, assuming a queue size of, say
100,000? Any information appreciated.
- Oliver.
"Austad, Jay" wrote:
> Non-unique emails will most likely be generated by other machines and send
> the box running mini-qmail via smtp. Non-unique emails will be a small
> percentage of what gets sent out, for now.
>
> Jay
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
> Sent: Saturday, July 15, 2000 12:10 AM
> To: '[EMAIL PROTECTED]'
> Subject: Re: questions about performance and setup
>
> On Fri, Jul 14, 2000 at 07:01:46PM -0500, Austad, Jay wrote:
> > >Then have the script that does the mailing call randomly
> > >on of the /var/qmail#/bin/qmail-inject. This will emulate round robin
> > >without any patching.
> >
> > Won't this way be a performance hit though? I admit, it is an easy
> solution
>
> No. My experience is that the cost of running a script to inject the mail
> in a way similar to that mentioned above, is pretty small compared to the
> queue injection cost and the delivery cost. sh or perl will be fine.
>
> > and would work excellent, but I have to think about efficiency also. C
> code
> > is much faster than shell or perl, and I'd like to set it up once and not
> > have to ever worry about again, or at least for a long, long time.
> >
> > As I said, we're doing 50 million emails a month right now, but this is
> > increasing substantially each month, and as we rollout new subscription
> > services, we'll have even more load. Sending 10 times this amount by the
> > same time next year is a good possibility, possibly sooner as we seem to
> > underestimate the rate at which we're growing much of the time...
>
> You may also need to look at the scalability of the generation of the
> emails. One system I recently looked at claimed to be able to generate
> nicely unique emails at a targetted database, but it burned CPU like
> it was free - just in generating the content.
>
> Mark.
>
> >
> > Jay
> >
> > -----Original Message-----
> > From: JuanE [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, July 14, 2000 5:55 PM
> > To: '[EMAIL PROTECTED]'
> > Subject: Re: questions about performance and setup
> >
> >
> >
> > Jay,
> >
> > That's the beauty of having multiple instances, not having to patch qmail.
> > All you need to do is install qmail once per machine (ie, /var/qmail1,
> > /var/qmail2,...). Then have the script that does the mailing call randomly
> > on of the /var/qmail#/bin/qmail-inject. This will emulate round robin
> > without any patching.
> >
> > JES
> >
> > Austad, Jay writes:
> >
> > > Where would I start in the code to modify the QMQP servers list so that
> it
> > > would load balance between all of the servers in the list instead of
> just
> > > using the first one it can contact? This would be very useful to me. I
> > > assume qmail-qmqpc.c is one of them, are there others I would need to
> play
> > > around with?
> > >
> > > Jay
> > >
> > > -----Original Message-----
> > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
> > > Sent: Friday, July 14, 2000 3:55 PM
> > > To: '[EMAIL PROTECTED]'
> > > Subject: Re: questions about performance and setup
> > >
> > >
> > > On Fri, Jul 14, 2000 at 02:29:06PM -0500, Austad, Jay wrote:
> > > > I already have Mandrake Linux 7.0 and 7.1 running on multiple Dell
> boxes
> > > > with no trouble, some of them took work to get going, but it runs
> well.
> > I
> > > > have a few Crystal PC's here also that I may use instead, dual PIII
> > 550's
> > > > with 512MB ram and 9 or 18GB 10000rpm drives. I'll probably use these
> > for
> > > > testing.
> > >
> > > I agree with the earlier poster that more spindles for your queue
> > > (c/- raid) is a good thing in general.
> > >
> > > > The bulk of the messages will be the same content to many rcpt's.
> > > However,
> > > > once in awhile we'll have 100,000 different messages go out to 100,000
> > > > different people.
> > > >
> > > > Since the QMQP support under mini-qmail doesn't load balance, can I
> feed
> > > it
> > > > a hostname with multiple dns entries (round-robin dns)? Or better
> yet,
> > > how
> > > > easy would it be to modify the qmail code to just load balance between
> > > them?
> > >
> > > The manpage for qmail-qmqpc tells us that they have to be IP addresses
> > > in qmqpservers so a RR DNS won't help. If all of the messages are
> > generated
> > > on one machine, then I'd be inclined to go for a much simpler solution
> > > than modifying qmail. I'd have an instance of qmail for each outbound
> > > server with the appropriate qmqpservers entry, then have your queue
> > > insertion script do a round-robin itself by simply cycling thru
> > > the qmail-inject command associated with each instance.
> > >
> > > for instance in 1 2 3 4 5
> > > do
> > > getnext_message_details()
> > > /var/qmail{$instance}/bin/qmail-inject currentmessage .... details
> > > done
> > >
> > > Or some such.
> > >
> > >
> > > Alternatively, if you have money to burn, maybe a layer four switch
> > > with load-balancing skills.
> > >
> > >
> > > Mark.
> > >
> > >
> > > >
> > > > Jay
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
> > > > Sent: Friday, July 14, 2000 2:09 PM
> > > > To: '[EMAIL PROTECTED]'
> > > > Subject: Re: questions about performance and setup
> > > >
> > > >
> > > > > Here's what I need to know:
> > > > >
> > > > > 1. How well does qmail take advantage of multiple processors? How
> > much
> > > >
> > > > Indreectly, quite well as it forks many processes, thus if the OS
> takes
> > > > good advantage of your CPUs, then qmail inherits that advantage.
> > > >
> > > > > memory and disk will I need? (we're at 50 million messages per
> month
> > > now,
> > > >
> > > > Are these message unique per target address or the same. If unique,
> your
> > > > requirements are vastly different and very queue/disk intensive. If
> they
> > > > are the same and you take advantage or VERP support on qmail, then
> > > > your load will mainly be sending related which will benefit from
> > > > more memory, multiple instances, etc.
> > > >
> > > > > and we only send out monday-friday, so that's over 2 million
> messages
> > > per
> > > > > day, and it's only going up)
> > > > >
> > > > > 2. How many messages per day would one estimate that each of these
> > > > servers
> > > > > could do?
> > > > >
> > > > > 3. I read about mini-qmail and how it's about 100 times faster
> > blasting
> > > > out
> > > > > email to QMQP servers. Since you can specify multiple QMQP servers,
> > if
> > > I
> > > > > have a fourth machine running mini-qmail and managing the actual
> > mailing
> > > > > list, can I add the other 3 as QMQP servers and have it load balance
> > > > between
> > > > > all 3 for sending out mail? (this way I could add more servers
> easily
> > > if
> > > > I
> > > > > needed to)
> > > >
> > > > The qmqp support doesn't load balance. It simply takes the first one
> > > > it can connect to.
> > > >
> > > > > 4. Can I easily make qmail run an external script for each bounced
> > mail?
> > > >
> > > > Absolutely.
> > > >
> > > > > 5. Anything else I should know?
> > > >
> > > > That all hinges on whether your emails are unique for each recipient
> or
> > > > not. Or more importantly, the average number of recipients per unique
> > > > email.
> > > >
> > > >
> > > > Regards.
> > >
> >
> >
