Re: Multiple instances of qmail...
[EMAIL PROTECTED] writes: So let's say each cluster node was using something like /var/qmail/queue/_NODE_IP_HERE_/ on the NFS server, it wouldn't be a problem for the delivery or the Maildirs? The docs fairly clearly say that putting the queue on NFS is a no-no. My understanding has always been that this isn't just because you might be tempted to share it, which obviously wouldn't work, but that the semantics of NFS are not sufficient for how the qmail-queue and delivery programs are written. From 'THOUGHTS', as distributed with qmail-1.03: 5. Handling queued mail (qmail-send, qmail-clean) The queue directory must be local. Mounting it over NFS is extremely dangerous---not that this stops people from running sendmail that way! Diskless hosts should use mini-qmail instead. Delivery to Maildirs on NFS is fine - the order of operations specified are constructed in order that it can be "safe" - although you should note that the recommendation is still to deliver locally and read remotely if you must (obviously on a dedicated NFS server you can't deliver locally.) James.
Re: Multiple instances of qmail...
On Thu, Jan 18, 2001 at 09:12:02PM -0500, [EMAIL PROTECTED] wrote: Henning Brauer wrote: On Thu, Jan 18, 2001 at 07:35:33PM -0500, [EMAIL PROTECTED] wrote: My question is, is it possible to run multiple instances of qmail, sharing the same disk structure, configuration, etc.. at least /var/qmail/queue/ must be local. Maildirs on NFS isn't that problem. I won't rely on linux NFS, but thats another thing. So let's say each cluster node was using something like /var/qmail/queue/_NODE_IP_HERE_/ on the NFS server, it wouldn't be a problem for the delivery or the Maildirs? I said /var/qmail/queue must be local, and I mean local. local is definitely another thing as "over NFS". I've taken a quick look at the qmail-smtpd man pages, If we are talking about the queue you should look at qmail-queue of course. As for the OS issue, I've had nothing but trouble with *BSDs, and where You made a fault somewhere. What I'm after is not really a performance boost from qmail (I'm sure it could run as a dedicated server and perform beyond expectations), it's more of a high availability issue. I'm planning on using a network block device on the NFS HA and NFS in one sentence. Interesting. server to do a network RAID-1 for everything that is written to the server, with heartbeat to automatically take over if the main server fails. Coupled with the hardware raid,journaling filesystem, and the clustered servers, I think it would be pretty hard to bring the system completely down. (knock on head) Buy two machines with _fast_ local disk storage systems and place the Maildirs on a shared RAID subsystem. NFS isn't suited for HA. If so, wouldn't this indicate that there aren't any concurrency issue with the queue? Disk throughput for the queue disk is the first performance problem on heavy loaded qmail systems, given you have enough Bandwidth. The queue relies on a good filesystem, the inode stuff is _very_ important. FFS seems to fit best for qmail's needs. -- Henning Brauer | BS Web Services Hostmaster BSWS| Roedingsmarkt 14 [EMAIL PROTECTED] | 20459 Hamburg http://www.bsws.de | Germany
Re: Multiple instances of qmail...
[EMAIL PROTECTED] wrote: My question is, is it possible to run multiple instances of qmail, sharing the same disk structure, configuration, etc.. No. The queue cannot be shared by multiple instances of qmail. -Dave
Re: Multiple instances of qmail...
On Fri, 19 Jan 2001, Henning Brauer wrote: On Thu, Jan 18, 2001 at 07:35:33PM -0500, [EMAIL PROTECTED] wrote: My question is, is it possible to run multiple instances of qmail, sharing the same disk structure, configuration, etc.. at least /var/qmail/queue/ must be local. Maildirs on NFS isn't that problem. I won't rely on linux NFS, but thats another thing. And the queue is a point of failure, if the filesystem with the queue dies with mail in the queue, said mail is gone. Which means if you really want HA, the queue should sit on a RAID1 or similar. RAID5 would be a nasty performance hit. That's plan A. Plan B is setting up a dedicated mail server, which I would like to avoid if necessary, because it won't scale nearly as well. For first: for performance reasons one qmail machine _might_ be enough. You also might want to look at qmail-ldap with its native clustering support. http://www.nrg4u.com, docs at http://www.lifewithqmail.org/ldap/ You should also spend some thoughts on your OS, the *BSDs may be much better choices for a high volume qmail server. qmail-ldap looks great, LDAP lets you distribute your authorization information and supports HA similar to the way DNS works. However, qmail-ldap's definition of clustering does NOT include HA or failover, last I heard. It does handle distributing mail delivery across a cluster of machines in a transparent manner. $.02 Bill Carlson -- Systems Programmer[EMAIL PROTECTED]| Opinions are mine, Virtual Hospital http://www.vh.org/| not my employer's. University of Iowa Hospitals and Clinics|
Re: Multiple instances of qmail...
Dave Sill [EMAIL PROTECTED] writes: [EMAIL PROTECTED] wrote: My question is, is it possible to run multiple instances of qmail, sharing the same disk structure, configuration, etc.. No. The queue cannot be shared by multiple instances of qmail. OTOH, everything else (binaries, configuration, addresses) can be shared. Then if one queue disk dies, you've lost any mail that was in it, but other mail will be unaffected. paul
Re: Multiple instances of qmail...
On Fri, Jan 19, 2001 at 10:14:09AM -0600, Bill Carlson wrote: On Fri, 19 Jan 2001, Henning Brauer wrote: On Thu, Jan 18, 2001 at 07:35:33PM -0500, [EMAIL PROTECTED] wrote: My question is, is it possible to run multiple instances of qmail, sharing the same disk structure, configuration, etc.. at least /var/qmail/queue/ must be local. Maildirs on NFS isn't that problem. I won't rely on linux NFS, but thats another thing. And the queue is a point of failure, if the filesystem with the queue dies with mail in the queue, said mail is gone. Which means if you really want HA, the queue should sit on a RAID1 or similar. RAID5 would be a nasty performance hit. I took this a precedence, of course. That's plan A. Plan B is setting up a dedicated mail server, which I would like to avoid if necessary, because it won't scale nearly as well. For first: for performance reasons one qmail machine _might_ be enough. You also might want to look at qmail-ldap with its native clustering support. http://www.nrg4u.com, docs at http://www.lifewithqmail.org/ldap/ You should also spend some thoughts on your OS, the *BSDs may be much better choices for a high volume qmail server. qmail-ldap looks great, LDAP lets you distribute your authorization information and supports HA similar to the way DNS works. However, qmail-ldap's definition of clustering does NOT include HA or failover, last I heard. It does handle distributing mail delivery across a cluster of machines in a transparent manner. You are right. qmail-ldap's clustering is just one important piece of you HA solution. Use some kind of IP based HA solution (like BigIP) for the MXes and the user intercating machines (outgoing SMTP, pop/imap), and build a hot standby solution for your Maildir holding machine. An external RAID with at least two external SCSI busses is a must here. $.02 Bill Carlson -- Systems Programmer[EMAIL PROTECTED] | Opinions are mine, Virtual Hospital http://www.vh.org/| not my employer's. University of Iowa Hospitals and Clinics | -- Henning Brauer | BS Web Services Hostmaster BSWS| Roedingsmarkt 14 [EMAIL PROTECTED] | 20459 Hamburg http://www.bsws.de | Germany
Re: Multiple instances of qmail...
Exactly.. I have 7 instances of qmail running on a Linux box (don't ask why its a rather long complicated story) and has been that way for about a year now.. No lost messages or anything. Just make sure your running the big concurrency patch. Other than that as far as performance .. It does work faster when each of our major clients has his own qmail process to goto. However that's about the only benefit. --JT - Original Message - From: "Paul Jarc" [EMAIL PROTECTED] To: "Dave Sill" [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Friday, January 19, 2001 8:30 AM Subject: Re: Multiple instances of qmail... Dave Sill [EMAIL PROTECTED] writes: [EMAIL PROTECTED] wrote: My question is, is it possible to run multiple instances of qmail, sharing the same disk structure, configuration, etc.. No. The queue cannot be shared by multiple instances of qmail. OTOH, everything else (binaries, configuration, addresses) can be shared. Then if one queue disk dies, you've lost any mail that was in it, but other mail will be unaffected. paul
Re: Multiple instances of qmail...
[EMAIL PROTECTED] writes: Henning Brauer wrote: On Thu, Jan 18, 2001 at 07:35:33PM -0500, [EMAIL PROTECTED] wrote: My question is, is it possible to run multiple instances of qmail, sharing the same disk structure, configuration, etc.. at least /var/qmail/queue/ must be local. Maildirs on NFS isn't that problem. I won't rely on linux NFS, but thats another thing. So let's say each cluster node was using something like /var/qmail/queue/_NODE_IP_HERE_/ on the NFS server, it wouldn't be a problem for the delivery or the Maildirs? If this is the case, is there a startup option I can give to qmail-smtpd to change it's mail queue path? (Or a simple hack) The simple hack is to make /var/qmail/queue a symlink to the real queue directory, somewhere on an NFS server. But unless your system is extremely small, NFS performance will be an issue. We were not able to use NFS-mounted queues for a 40,000 user system at all, with Solaris clients and a NetApp server, arguably two of the best NFS implementations out there. -ScottG.
Multiple instances of qmail...
Hi folks, I'm setting up a high availability system, and have a question on the way that qmail handles disk writes. I'll be using LVS (linux virtual server) patch to the linux kernel coupled with heartbeat/ldirector to do load balancing and high availability on a farm of computers which will boot off a cd-rom, and mount their root partition via NFS. The cluster nodes will all have internal ip addresses, and the monitoring machines will do ip tunneling to make these machines appear as one single machine, serving thousands of domains. The NFS server will be the where all the user data is held, on a raid 1+0 disk array with ReiserFS, apache, qmail, /home, etc... (It will be running linux 2.2.18 or 2.4.X, kernel nfs server, lots of RAM) My question is, is it possible to run multiple instances of qmail, sharing the same disk structure, configuration, etc.. Basically, have each cluster node run qmail and keep all the data on one server without encountering inconsistencies in file locking (assuming the latest version of knfsd does not have any locking issues). That's plan A. Plan B is setting up a decicated mail server, which I would like to avoid if necessary, because it won't scale nearly as well. Thanks in advance, Mark Steele VP research and development Inet Technologies Inc. email: [EMAIL PROTECTED]
Re: Multiple instances of qmail...
On Thu, Jan 18, 2001 at 07:35:33PM -0500, [EMAIL PROTECTED] wrote: My question is, is it possible to run multiple instances of qmail, sharing the same disk structure, configuration, etc.. at least /var/qmail/queue/ must be local. Maildirs on NFS isn't that problem. I won't rely on linux NFS, but thats another thing. That's plan A. Plan B is setting up a decicated mail server, which I would like to avoid if necessary, because it won't scale nearly as well. For first: for performance reasons one qmail machine _might_ be enough. You also might want to look at qmail-ldap with its native clustering support. http://www.nrg4u.com, docs at http://www.lifewithqmail.org/ldap/ You should also spend some thoughts on your OS, the *BSDs may be much better choices for a high volume qmail server. -- Henning Brauer | BS Web Services Hostmaster BSWS| Roedingsmarkt 14 [EMAIL PROTECTED] | 20459 Hamburg http://www.bsws.de | Germany
Re: Multiple instances of qmail...
Henning Brauer wrote: On Thu, Jan 18, 2001 at 07:35:33PM -0500, [EMAIL PROTECTED] wrote: My question is, is it possible to run multiple instances of qmail, sharing the same disk structure, configuration, etc.. at least /var/qmail/queue/ must be local. Maildirs on NFS isn't that problem. I won't rely on linux NFS, but thats another thing. So let's say each cluster node was using something like /var/qmail/queue/_NODE_IP_HERE_/ on the NFS server, it wouldn't be a problem for the delivery or the Maildirs? If this is the case, is there a startup option I can give to qmail-smtpd to change it's mail queue path? (Or a simple hack) I've taken a quick look at the qmail-smtpd man pages, and haven't seen anything pertaining to that particular issue. I suppose it's hardcoded. That's plan A. Plan B is setting up a decicated mail server, which I would like to avoid if necessary, because it won't scale nearly as well. For first: for performance reasons one qmail machine _might_ be enough. You also might want to look at qmail-ldap with its native clustering support. http://www.nrg4u.com, docs at http://www.lifewithqmail.org/ldap/ You should also spend some thoughts on your OS, the *BSDs may be much better choices for a high volume qmail server. -- Henning Brauer | BS Web Services Hostmaster BSWS| Roedingsmarkt 14 [EMAIL PROTECTED] | 20459 Hamburg http://www.bsws.de | Germany I just took a look at the ldap solution, and I don't think it would fit in very well with the system I'm setting up. It looks cool but impractical. As for the OS issue, I've had nothing but trouble with *BSDs, and where they have failed me, linux hasn't :( (don't want to start an OS war here, just the facts in my scenario) What I'm after is not really a performance boost from qmail (I'm sure it could run as a dedicated server and perform beyond expectations), it's more of a high availability issue. I'm planning on using a network block device on the NFS server to do a network RAID-1 for everything that is written to the server, with heartbeat to automatically take over if the main server fails. Coupled with the hardware raid,journaling filesystem, and the clustered servers, I think it would be pretty hard to bring the system completely down. (knock on head) I've also been trying to find some documentation on this topic, and so far the only thing I found indicates that this is a non-issue, since tcpserver spawns a new qmail-smtpd for each incomming request. (an archive of some discussion list) If so, wouldn't this indicate that there aren't any concurrency issue with the queue? Thanks in advance Mark Steele VP research and development Inet Technologies Inc. [EMAIL PROTECTED]
Re: problems with multiple instances of qmail.
Thanks, I did found the error message "VFS: file-max limit 4096 reached" in dmesg. It's still weird. I have concurencyremote=250 for each instance, and the open files value obtained from ulimit is 1024 ( 1024 2*250 + 5). It seems to me there's enough margin for it. BTW, how is the requirement for filedescriptor 2*concurrencyremote + 5 working for mulitiple qmail instances? What do 'cat /proc/sys/fs/file-nr' and 'cat /proc/sys/fs/file-max' give you? Also adding up the number of files qmail qmail opens will give you an under estimate. Qmail is not the only thing running on the machine and file descripters are opened for other reasons than just explicit file access by an application. Try doing 'echo 8192 /proc/sys/fs/file-max' as root then a 'cat /proc/sys/fs/file-nr' after some time. The 3 numbers from file-nr are maximum used, currently in use and maximum available. -- Mark Evans St. Peter's CofE High School Phone: +44 1392 204764 X109 Fax: +44 1392 204763
Re: problems (under Linux) with multiple instances of qmail.
Dongping Deng writes: I have concurencyremote=250 for each instance, and the open files value obtained from ulimit is 1024 ( 1024 2*250 + 5). It seems to me there's enough margin for it. Your problem has nothing to do with per-process limits. The kernel has a table of 4096 ofiles shared by the entire system. That table is full. Bump it up to size 16384. If you do this then also take a look at the (in core) inode limit. Otherwise you could easily swap one limiting factor for another. Inode limit is /proc/sys/fs/inode-max, file descriptor limit is in /proc/sys/fs/file-max -- Mark Evans St. Peter's CofE High School Phone: +44 1392 204764 X109 Fax: +44 1392 204763
Re: problems with multiple instances of qmail.
On 13 Nov 1999, D. J. Bernstein wrote: Dongping Deng writes: I have concurencyremote=250 for each instance, and the open files value obtained from ulimit is 1024 ( 1024 2*250 + 5). It seems to me there's enough margin for it. Your problem has nothing to do with per-process limits. The kernel has a table of 4096 ofiles shared by the entire system. That table is full. Bump it up to size 16384. Ahhh. maybe a stupid question... How do you bump up the ofiles tables on solaris??
Re: problems with multiple instances of qmail.
Dongping Deng writes: I have concurencyremote=250 for each instance, and the open files value obtained from ulimit is 1024 ( 1024 2*250 + 5). It seems to me there's enough margin for it. Your problem has nothing to do with per-process limits. The kernel has a table of 4096 ofiles shared by the entire system. That table is full. Bump it up to size 16384. You should also try compiling qmail statically. This lowers memory use, ofile use, and CPU use on most systems. (It's impossible under Solaris, but Solaris will be slow no matter what you do.) ---Dan
RE: problems with multiple instances of qmail.
On Thu, Nov 11, 1999 at 05:15:14PM -0800, Dongping Deng wrote: If I only inject to one of them (both qmail/qmail2 are running), it also works fine. It seems the two instances start to do something funny. Any ideas? You're running out of filedescriptors (either for that users (qmail*) or system wide). Increase the limits and it should work. For that you either have to use "ulimit" before starting qmail, configure some kernel limits or even build a new kernel. Thanks, I did found the error message "VFS: file-max limit 4096 reached" in dmesg. It's still weird. I have concurencyremote=250 for each instance, and the open files value obtained from ulimit is 1024 ( 1024 2*250 + 5). It seems to me there's enough margin for it. BTW, how is the requirement for filedescriptor 2*concurrencyremote + 5 working for mulitiple qmail instances? dp
problems with multiple instances of qmail.
We just bought a valinux machine running redhat 6.0. I installed two instances (qmail and qmail2) of qmail. I wrote a testing script to inject mail (1000) alternatively to qmail and qmail2. I always I get these errors after a few hundreds messages are injected. (sometimes these) bin/qmail-queue: error in loading shared libraries: libc.so.6: cannot open shared object file: Error 23 qmail-inject: fatal: qq temporary problem (#4.3.0) bin/qmail-queue: error in loading shared libraries: libc.so.6: cannot open shared object file: Error 23 qmail-inject: fatal: qq temporary problem (#4.3.0) qmail-inject: fatal: unable to run qmail-queue Broken pipe (sometimes these) qmail-inject: fatal: unable to exec qq (#4.3.0) qmail-inject: fatal: qq crashed (#4.3.0) bin/qmail-queue: error in loading shared libraries: libc.so.6: cannot open shared object file: Error 23 qmail-inject: fatal: qq temporary problem (#4.3.0) bin/qmail-queue: error in loading shared libraries: libc.so.6: cannot open shared object file: Error 23 qmail-inject: fatal: qq temporary problem (#4.3.0) bin/qmail-queue: error in loading shared libraries: libc.so.6: cannot open shared object file: Error 23 qmail-inject: fatal: qq temporary problem (#4.3.0) bin/qmail-queue: error in loading shared libraries: libc.so.6: cannot open shared object file: Error 23 qmail-inject: fatal: qq temporary problem (#4.3.0) bin/qmail-queue: error in loading shared libraries: libc.so.6: cannot open shared object file: Error 23 qmail-inject: fatal: qq temporary problem (#4.3.0) bin/qmail-queue: error in loading shared libraries: libc.so.6: cannot open shared object file: Error 23 qmail-inject: fatal: qq temporary problem (#4.3.0) bin/qmail-queue: error in loading shared libraries: libc.so.6: cannot open shared object file: Error 23 qmail-inject: fatal: qq temporary problem (#4.3.0) bin/qmail-queue: error in loading shared libraries: libc.so.6: cannot open shared object file: Error 23 qmail-inject: fatal: qq temporary problem (#4.3.0) qmail-inject: fatal: unable to exec qq (#4.3.0) qmail-inject: fatal: qq crashed (#4.3.0) qmail-inject: fatal: qq crashed (#4.3.0) qmail-inject: fatal: qq crashed (#4.3.0) Broken pipe If I shut down one of them, say qmail2, run the same script, I don't get these error messages. Then after I restart qmail2, the injected msgs start to go out. If I only inject to one of them (both qmail/qmail2 are running), it also works fine. It seems the two instances start to do something funny. Any ideas? Thanks dp
Re: problems with multiple instances of qmail.
On Thu, Nov 11, 1999 at 05:15:14PM -0800, Dongping Deng wrote: If I only inject to one of them (both qmail/qmail2 are running), it also works fine. It seems the two instances start to do something funny. Any ideas? You're running out of filedescriptors (either for that users (qmail*) or system wide). Increase the limits and it should work. For that you either have to use "ulimit" before starting qmail, configure some kernel limits or even build a new kernel. \Maex -- SpaceNet GmbH | http://www.Space.Net/ | Yeah, yo mama dresses Research Development| mailto:[EMAIL PROTECTED] | you funny and you need Joseph-Dollinger-Bogen 14 | Tel: +49 (89) 32356-0| a mouse to delete files D-80807 Muenchen | Fax: +49 (89) 32356-299 |
