Re: High Availability on Postfix
Thegeswini S: Could some one help on below request ? On Thu, Apr 16, 2015 at 1:02 PM, Thegeswini S thegesw...@gmail.com wrote: Presently we use primary MTA as Postfix for outbound mail server and we were not configured inbound mails as we don;t have POP server in our env. The mail server resides on primary site and all the application servers including DR sites servers, uses this mail server as relay system and send mails to ISP... Now the requirement is to create an secondary outbound mail server on DR site, incase of Primay server or site is down. I would like to know any solution sending messages from backup outbound when primary is down ? If you mean: have systems on your corporate network send external email through the backup outbound mail server when the primary outbound mail server is down, then I recommend that you use DNS MX records, with the most-preferred records resolving to the primary mail server, and with the less-preferred records resolving to the secondary mail server. If your infrastructure does not use DNS MX records internally, then you can use A records instead, but then you have no preference feature. Otherwise, you need to find a solution that provides similar functionality. For example, a number of strategically-placed proxy servers (HAproxy, nginx) that direct clients to the best MTA. Wietse
Re: High Availability
On Sun, Apr 6, 2014 at 2:05 PM, Ramesh itsrames...@yahoo.co.in wrote: Hi All, Hi! Presently we have primary MX and backup MX servers, when primary goes down mails will be queued in secondary MX, once primary restored all messages pushed from backup MX to primary MX, messages are not lost. I would like to know any solution sending and receiving messages from backup MX when primary MX is down? Appreciate suggestion, recently due to major internet service down, we are not able to check mails or send mails. As Wietse already said, you can just have a replicated message store, as long as you accept that outgoing mail queue (most sites have some messages lying there, waiting to retry) and *maybe* one or two messages in the intermediate queues (highly unlikely) will be unavailable until you restore primary, and could be potentially lost if primary dies. Otherwise you would need to replicate queue directories, likely using DRBD. Yes, you can use DRBD over long-distance links, but you will have increased latency and reduced write performance (search for DRBD Proxy for an explanation). I have implemented DRBD using softlayer's private network, but only for systems where reads/writes ratio is high. Ildefonso.
Re: High Availability
On 7/4/2014 8:17 πμ, Patrick Ben Koetter wrote: On Linux use DRBD to replicate mail queues between a pair of machines and crm to control a second Postfix instance that will be started locally to pickup any remaining mails once the partner machine dies. Hmm, I think DRBD is only advised in cases where the net link between the replicated boxes is guaranteed and low-latency; so I guess probably this is not a working solution between different data centers as discussed here. However, I can't suggest alternatives, I am afraid... My instinct (though not always correct :-) ) tells me also that near-real-time file sync (like using lsyncd with rsync) should not be a suggested solution for queue replication. One could investigate whether Apache Helix (http://helix.apache.org/) can be a viable solution. All the best, Nick
Re: High Availability
Miles Fidelman: To find solutions, open your favorite search engine and try cyrus mailbox replication, dovecot meailbox replication, and so on. I've been wondering about this too, and it strikes me that mailbox replication is only relevant to local delivery. What about replicating the various intermediate mail queues? (My current HA setup is brute force - a failover virtual machine, with a completely replicated file system. But I've been looking for ways that are more granular, and that are easier to do across two separate data centers.) Have you considered the following: - Inbound mail spends a fraction of a second in the queue. - Inbound mail spends days or weeks or more in the mailbox. - If an MTA goes down, mail flows via alternate MX hosts. - If the mailbox store goes down, then you have no mail. That's why high availability focuses on the mailbox store, not on the MTA in the middle. Wietse
Re: High Availability
Wietse Venema wrote: Miles Fidelman: To find solutions, open your favorite search engine and try cyrus mailbox replication, dovecot meailbox replication, and so on. I've been wondering about this too, and it strikes me that mailbox replication is only relevant to local delivery. What about replicating the various intermediate mail queues? (My current HA setup is brute force - a failover virtual machine, with a completely replicated file system. But I've been looking for ways that are more granular, and that are easier to do across two separate data centers.) Have you considered the following: - Inbound mail spends a fraction of a second in the queue. - Inbound mail spends days or weeks or more in the mailbox. - If an MTA goes down, mail flows via alternate MX hosts. - If the mailbox store goes down, then you have no mail. That's why high availability focuses on the mailbox store, not on the MTA in the middle. Well yes, in theory - but in practice we run a bunch of email lists, and I find that there are always cases where one or more destinations are temporarily unavailable - so there are various messages that will hang around for a while. So HA for the queues is not unreasonable to think about. Miles -- In theory, there is no difference between theory and practice. In practice, there is. Yogi Berra
Re: High Availability
Miles Fidelman: Have you considered the following: - Inbound mail spends a fraction of a second in the queue. - Inbound mail spends days or weeks or more in the mailbox. - If an MTA goes down, mail flows via alternate MX hosts. - If the mailbox store goes down, then you have no mail. That's why high availability focuses on the mailbox store, not on the MTA in the middle. [talking about OUTBOUND mail which was not the subject of this thread] You change the topic of the discussion and then claim some contradiction. Wietse
Re: High Availability
Wietse Venema wrote: Miles Fidelman: Have you considered the following: - Inbound mail spends a fraction of a second in the queue. - Inbound mail spends days or weeks or more in the mailbox. - If an MTA goes down, mail flows via alternate MX hosts. - If the mailbox store goes down, then you have no mail. That's why high availability focuses on the mailbox store, not on the MTA in the middle. [talking about OUTBOUND mail which was not the subject of this thread] You change the topic of the discussion and then claim some contradiction. Not to be argumentative or anything, but... original query was: Presently we have primary MX and backup MX servers, when primary goes down mails will be queued in secondary MX, once primary restored all messages pushed from backup MX to primary MX, messages are not lost. I would like to know any solution sending and receiving messages from backup MX when primary MX is down? Which sure looks like it includes outbound (sending) as part of the topic. Miles
Re: High Availability
Miles Fidelman: Wietse Venema wrote: Miles Fidelman: Have you considered the following: - Inbound mail spends a fraction of a second in the queue. - Inbound mail spends days or weeks or more in the mailbox. - If an MTA goes down, mail flows via alternate MX hosts. - If the mailbox store goes down, then you have no mail. That's why high availability focuses on the mailbox store, not on the MTA in the middle. [talking about OUTBOUND mail which was not the subject of this thread] You change the topic of the discussion and then claim some contradiction. Not to be argumentative or anything, but... original query was: Presently we have primary MX and backup MX servers, when primary goes down mails will be queued in secondary MX, once primary restored all messages pushed from backup MX to primary MX, messages are not lost. I He describes the flow of email for domains that have MX records with the names of his MX hosts (when the primary MX is down, mail queues on the secondary MX, from which it's sent to the primary). In other words, he describes inbound email. Outbound mail. on the other hand, is sent to the remote MX hosts of remote destination domains. Those remote MX hosts are not the MX hosts that he is talking about. Wietse
Re: High Availability
I don't see any reason to complicate things by implementing HA solution, when you can simply have multiple MX records. On Mon, Apr 7, 2014 at 12:35 AM, Ramesh itsrames...@yahoo.co.in wrote: Hi All, Presently we have primary MX and backup MX servers, when primary goes down mails will be queued in secondary MX, once primary restored all messages pushed from backup MX to primary MX, messages are not lost. I would like to know any solution sending and receiving messages from backup MX when primary MX is down? Appreciate suggestion, recently due to major internet service down, we are not able to check mails or send mails. Thanks Ramesh
Re: High Availability
Ramesh: Presently we have primary MX and backup MX servers, when primary goes down mails will be queued in secondary MX, once primary restored all messages pushed from backup MX to primary MX, messages are not lost. I would like to know any solution sending and receiving messages from backup MX when primary MX is down? This is the wrong question on the wrong mailing list. The right question is I need a message store that is replicated in multiple locations. Once you have such a message store, any number of Postfix MTAs and mail clients can use it. Wietse
Re: High Availability
Im sorry Wietse, Please let me know how to implement this, share me urls i will go through it how replication helps to solve our problem. Regards, Ramesh On Monday, 7 April 2014 1:00 AM, Wietse Venema wie...@porcupine.org wrote: Ramesh: Presently we have primary MX and backup MX servers, when primary goes down mails will be queued in secondary MX, once primary restored all messages pushed from backup MX to primary MX, messages are not lost. I would like to know any solution sending and receiving messages from backup MX when primary MX is down? This is the wrong question on the wrong mailing list. The right question is I need a message store that is replicated in multiple locations. Once you have such a message store, any number of Postfix MTAs and mail clients can use it. Wietse
Re: High Availability
Wietse: The right question is I need a message store that is replicated in multiple locations. Once you have such a message store, any number of Postfix MTAs and mail clients can use it. Ramesh: Please let me know how to implement this, share me urls i will go through it how replication helps to solve our problem. To find solutions, open your favorite search engine and try cyrus mailbox replication, dovecot meailbox replication, and so on. Wietse A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail?
Re: High Availability
Hi Wietse, Wietse Venema wrote: Wietse: The right question is I need a message store that is replicated in multiple locations. Once you have such a message store, any number of Postfix MTAs and mail clients can use it. Ramesh: Please let me know how to implement this, share me urls i will go through it how replication helps to solve our problem. To find solutions, open your favorite search engine and try cyrus mailbox replication, dovecot meailbox replication, and so on. I've been wondering about this too, and it strikes me that mailbox replication is only relevant to local delivery. What about replicating the various intermediate mail queues? (My current HA setup is brute force - a failover virtual machine, with a completely replicated file system. But I've been looking for ways that are more granular, and that are easier to do across two separate data centers.) Thanks, Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. Yogi Berra
Re: High Availability
* Miles Fidelman mfidel...@meetinghouse.net: Hi Wietse, Wietse Venema wrote: Wietse: The right question is I need a message store that is replicated in multiple locations. Once you have such a message store, any number of Postfix MTAs and mail clients can use it. Ramesh: Please let me know how to implement this, share me urls i will go through it how replication helps to solve our problem. To find solutions, open your favorite search engine and try cyrus mailbox replication, dovecot meailbox replication, and so on. I've been wondering about this too, and it strikes me that mailbox replication is only relevant to local delivery. What about replicating the various intermediate mail queues? (My current HA setup is brute force - a failover virtual machine, with a completely replicated file system. But I've been looking for ways that are more granular, and that are easier to do across two separate data centers.) On Linux use DRBD to replicate mail queues between a pair of machines and crm to control a second Postfix instance that will be started locally to pickup any remaining mails once the partner machine dies. p@rick -- [*] sys4 AG https://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Re: High Availability Solution for Postfix SMTP Server
From: Ansgar Wiechers li...@planetcobalt.net On 2012-06-18 Kaushal Shriyan wrote: Are there any High Availability Solution for Postfix SMTP Server meaning primary and secondary nodes in Active/Active or Active/Passive Clustering mode? Please describe the problem you're trying to solve instead of what you perceive as the solution. I will try to describe his problem: - Postfix server crashes in flame = problem. - Any (semi)-automated fallback server solution? Hence the High Availability Solution question... JD
Re: High Availability Solution for Postfix SMTP Server
John Doe wrote: From: Ansgar Wiechers li...@planetcobalt.net On 2012-06-18 Kaushal Shriyan wrote: Are there any High Availability Solution for Postfix SMTP Server meaning primary and secondary nodes in Active/Active or Active/Passive Clustering mode? Please describe the problem you're trying to solve instead of what you perceive as the solution. I will try to describe his problem: - Postfix server crashes in flame = problem. - Any (semi)-automated fallback server solution? Hence the High Availability Solution question... Well, there's the obvious one, which I use: Simply run Postfix on top of a high availability virtual machine stack. In my case: - replicated disks using drbd - Xen virtual machines - crm for failover management All the data is mirrored across two machines. If either a VM or an entire machine crashes, the VM simply restarts on the other node. If you wanted, you could set up Postfix as a managed resource, so that if only the postfix processes die, it gets restarted. You could probably use drbd and crm to mirror data and failover postfix w/o a VM in the middle. Take a look at http://www.linux-ha.org/wiki/Main_Page as a starting point. Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. Yogi Berra
Re: High Availability Solution for Postfix SMTP Server
John Doe: From: Ansgar Wiechers li...@planetcobalt.net On 2012-06-18 Kaushal Shriyan wrote: Are there any High Availability Solution for Postfix SMTP Server meaning primary and secondary nodes in Active/Active or Active/Passive Clustering mode? Please describe the problem you're trying to solve instead of what you perceive as the solution. I will try to describe his problem: - Postfix server crashes in flame = problem. - Any (semi)-automated fallback server solution? Hence the High Availability Solution question... Postfix persists all transactions to the file system. Thus, you'll need to make the file system highly-available. Techniques for doing that are outside the scope of Postfix. They just need to provide the same persistence guarantees (fsync(2), etc.) as a local disk. Wietse
Re: High Availability Solution for Postfix SMTP Server
On Jun 19, 2012, at 16:36, John Hudak wrote: He stated his requirement, specifically, the need for a high availability system. The details of what lead him to having this requirement are somewhat irrelevant - unless you want to go down the path of eliciting all the quality attributes and look at architectural tradeoffs (which is not what he is asking for). He wants some guidance on architectures for high availability postfix services. He did not state his availability requirements, e.g. 0.9, 0.99, 0.999 etc, but he doesn't have to. This is, if you will, his requirement. The solutions are the many architectural approaches that would be discussed in this, and other forums. I absolutely agree. With Ansgar, that is. Certain types of questions pop up again and again, and a lot of the problems that people bring to this list tend to originate in an incomplete (or incorrect) understanding of the software, its scope, and the protocols involved. Describing the problem instead the perceived solution will allow feedback on your assumptions, and it never hurts to have those fact checked by the many experienced people on this list. Between them, they operate everything from mom-and-cat home servers to large scale clusters, and chances are that at least one of them has already solved the same problem before. Describing your problem is simply the best way to tap into this list's potential. It'll get you better answers, faster. Cya, Jona -- On Mon, Jun 18, 2012 at 2:22 PM, Ansgar Wiechers li...@planetcobalt.net wrote: On 2012-06-18 Kaushal Shriyan wrote: Are there any High Availability Solution for Postfix SMTP Server meaning primary and secondary nodes in Active/Active or Active/Passive Clustering mode? Please describe the problem you're trying to solve instead of what you perceive as the solution. Regards Ansgar Wiechers -- Abstractions save us time working, but they don't save us time learning. --Joel Spolsky
Re: High Availability Solution for Postfix SMTP Server
On 2012-06-18 Kaushal Shriyan wrote: Are there any High Availability Solution for Postfix SMTP Server meaning primary and secondary nodes in Active/Active or Active/Passive Clustering mode? Please describe the problem you're trying to solve instead of what you perceive as the solution. Regards Ansgar Wiechers -- Abstractions save us time working, but they don't save us time learning. --Joel Spolsky
Re: high-availability configurations?
On 03/14/2012 04:19 PM, Miles Fidelman wrote: Hi Folks, I'm currently running a pretty basic high-availability configuration for our mail server (postfix) - it simply runs in a Xen virtual machine, with mirrored disks across two machines (DRBD), and failover of the VM if something goes wrong (pacemaker). I'm thinking about migrating the failover host to a 2nd datacenter - which makes disk mirroring and VM migration a bit trickier, and I really don't like how brittle all that infrastructure is, so I'm starting to think about application layer redundancy - two mailservers, at remote locations, multiple DNS records, and doing something to replicate ques, configurations, and local delivery. The goal is the same: keep processing mail if a machine goes down, and don't lose any data to machine or disk crashes. Which leads to a question: Are any of you running such a configuration? If so, can you describe what you're doing? And.. are there any good references, presentations, etc. that anybody knows about re. building high-availability, scalable, distributed mail processing infrastructure? Thank you very much, Miles Fidelman SMTP is designed to be redundant from the ground up; that's why you have multiple MX records. Any reasonable arguments why just running multiple MTAs does not work for you ? -- J.
Re: high-availability configurations?
Jeroen Geilman wrote: On 03/14/2012 04:19 PM, Miles Fidelman wrote: Hi Folks, I'm currently running a pretty basic high-availability configuration for our mail server (postfix) - it simply runs in a Xen virtual machine, with mirrored disks across two machines (DRBD), and failover of the VM if something goes wrong (pacemaker). I'm thinking about migrating the failover host to a 2nd datacenter - which makes disk mirroring and VM migration a bit trickier, and I really don't like how brittle all that infrastructure is, so I'm starting to think about application layer redundancy - two mailservers, at remote locations, multiple DNS records, and doing something to replicate ques, configurations, and local delivery. The goal is the same: keep processing mail if a machine goes down, and don't lose any data to machine or disk crashes. Which leads to a question: Are any of you running such a configuration? If so, can you describe what you're doing? And.. are there any good references, presentations, etc. that anybody knows about re. building high-availability, scalable, distributed mail processing infrastructure? Thank you very much, Miles Fidelman SMTP is designed to be redundant from the ground up; that's why you have multiple MX records. Any reasonable arguments why just running multiple MTAs does not work for you ? Machines crash while mail is being processed. Disks crash and take queues with them. With our current approach, we rely on redundancy at the virtual machine and filesystem level. I'd rather push things up to the application layer. -- In theory, there is no difference between theory and practice. In practice, there is. Yogi Berra
Re: high-availability configurations?
On Wed, Mar 14, 2012 at 10:49 AM, Miles Fidelman mfidel...@meetinghouse.net wrote: Hi Folks, Hi. I'm currently running a pretty basic high-availability configuration for our mail server (postfix) - it simply runs in a Xen virtual machine, with mirrored disks across two machines (DRBD), and failover of the VM if something goes wrong (pacemaker). I'm thinking about migrating the failover host to a 2nd datacenter - which makes disk mirroring and VM migration a bit trickier, and I really don't like how brittle all that infrastructure is, so I'm starting to think about application layer redundancy - two mailservers, at remote locations, multiple DNS records, and doing something to replicate ques, configurations, and local delivery. The goal is the same: keep processing mail if a machine goes down, and don't lose any data to machine or disk crashes. Which leads to a question: Are any of you running such a configuration? If so, can you describe what you're doing? And.. are there any good Well, first question here: how much traffic are you going to handle? And now, my experience (please, postfix-list purists, stop reading now, this is more related to DRBD than it is to postfix): I have a HA cluster with two nodes on two locations, on softlayer, due that softlayer provides unlimited inter-server connectivity (please, if someone knows another hosting company that does this -unlimited communication between servers in different DCs-, let me know: softlayer is quite expensive), I'm just using the private network (that use to run at 200~500Mbps) to replicate the DRBD volume. I had several issues, but I suggest you try, and then post on the corresponding lists (DRBD, pacemaker, corosync, heartbeat, ). I have VM-level failover here, but it is pretty much the same to setup service-level failover. About multiple DNS records, etc... I just used low TTL DNS, and a dynamic DNS setup, so that the VM updates the DNS record on failover. On a side note: I personally believe that service-level HA configuration is better than VM-level. references, presentations, etc. that anybody knows about re. building high-availability, scalable, distributed mail processing infrastructure? You can use postfix's mail routing capabilities to have distributed mail processing, ie: have some users on one server, and others at the other server... it is neat. Sincerely, Ildefonso Camargo
Re: high-availability mail cluster?
On Fri, 21 Oct 2011 09:27:03 -0400, Miles Fidelman mfidel...@meetinghouse.net wrote: Hi Folks, I'm about to rebuild a server farm, and I'm thinking about alternate approaches to high-availability for our mail services. Right now, I just run a collection of services (including mail) on a virtual machine, on top of a disk farm, with auto-failover to a hot-spare backup on a 2nd machine. With the addition of a few more machines into the rack, I'm thinking about dis-aggregating several services, and wondering about service-specific high-availability strategies. Can anybody point me to examples, howtos, or what have you on building a high-availability mail cluster - I'm running Postfix, Amavisd, Spammassassin, ClamAV (plus Sympa for list management, and UW IMAP - but those aren't technically part of the mail processing). I'm specifically looking for approaches to redundant storage of mail ques, failover models, recovery from failure, and so forth. Thanks much, Miles Fidelman Miles, I can outline our setup, which we have spent some time building. We have an external spam filtering solution, which in itself is load balanced and highly available which I will leave out. At the most basic layer we use VMware on commodity hardware, which gives us a lot of flexibility in deploying mail servers (i.e. cloning templating). If you're not already virtualized I would highly recommend it. In our environment we have four separate mail servers that perform a single task each: - Inbound email (postfix) - POP/IMAP (dovecot) - Outbound email (postfix dovecot for SASL) - Webmail (Roundcube) We do this to mitigate disaster risk, where if one service bugged out and caused the load to skyrocket, it would not break the other services. This setup has saved us from full blown outages many times, and instead we had a smaller outage that might have just impacted POP/IMAP or inbound mail separately. We also feel this solution is easier to manage as the configs are completely separated. We also have created two to four of each type of server, which are load balanced appropriately. We use NFS storage to tie all of the disk bound services together, as it is naturally a clustering solution for storage, and works great for this mail system. For super high availability we have implemented a NetApp Metro Cluster NAS, where we can instantly failover NFS services from our primary data center to our secondary data center 10 miles away. The virtual machines that host mail are located at both sites as well, and are clustered using a pair of hardware load balancers using VRRP for connectivity failover. This means close to zero downtime, which is really amazing. All of the authentication and aliasing is done from a Galera mysql database. Galera is a multi-master synchronous replication service for mysql, which allows us to host read/write capable cluster of mysql servers that exist at both data centers, providing the most crucial part of the HA solution in my mind. If you have any questions feel free to ask.
Re: high-availability mail cluster?
Am 21.10.2011 15:27, schrieb Miles Fidelman: Hi Folks, I'm about to rebuild a server farm, and I'm thinking about alternate approaches to high-availability for our mail services. Right now, I just run a collection of services (including mail) on a virtual machine, on top of a disk farm, with auto-failover to a hot-spare backup on a 2nd machine. With the addition of a few more machines into the rack, I'm thinking about dis-aggregating several services, and wondering about service-specific high-availability strategies. Can anybody point me to examples, howtos, or what have you on building a high-availability mail cluster - I'm running Postfix, Amavisd, Spammassassin, ClamAV (plus Sympa for list management, and UW IMAP - but those aren't technically part of the mail processing). I'm specifically looking for approaches to redundant storage of mail ques, failover models, recovery from failure, and so forth. Thanks much, Miles Fidelman i am using ha-loadbalancers, drbd-ocfs2 storage with dovecot clamav spamassassin postfix mysql on linux ubuntu lucid up to 5000 maildir mailboxes 2 lbs, 2 mail/imap/pop3servers, 1 central logging and backupserver nfs/syslog but thats only one choice, there are a lot of choices left which would work too -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: high-availability mail cluster?
Am 21.10.2011 18:10, schrieb Robert Schetterer: Am 21.10.2011 15:27, schrieb Miles Fidelman: Hi Folks, I'm about to rebuild a server farm, and I'm thinking about alternate approaches to high-availability for our mail services. Right now, I just run a collection of services (including mail) on a virtual machine, on top of a disk farm, with auto-failover to a hot-spare backup on a 2nd machine. With the addition of a few more machines into the rack, I'm thinking about dis-aggregating several services, and wondering about service-specific high-availability strategies. Can anybody point me to examples, howtos, or what have you on building a high-availability mail cluster - I'm running Postfix, Amavisd, Spammassassin, ClamAV (plus Sympa for list management, and UW IMAP - but those aren't technically part of the mail processing). I'm specifically looking for approaches to redundant storage of mail ques, failover models, recovery from failure, and so forth. Thanks much, Miles Fidelman i am using ha-loadbalancers, drbd-ocfs2 storage with dovecot clamav spamassassin postfix mysql on linux ubuntu lucid up to 5000 maildir mailboxes 2 lbs, 2 mail/imap/pop3servers, 1 central logging and backupserver nfs/syslog i just forgot, you may host this on vm machines too but thats only one choice, there are a lot of choices left which would work too i just forgot, you may host this on vm machines too -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
