Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Stefan G. Weichinger put forth on 11/2/2010 1:15 PM: A bit off-topic, sorry ... I want to set up a hot backup dovecot in a VM, aside the physical server, so I am very interested in the best practise to do so ... There isn't one. If there was Timo would have pointed you to the wiki. Doing server fail over is inherently problematic for a large number of reasons. The easiest way to implement it is to literally turn on the backup server (power on) when the primary fails. The backup comes up with the same hostname and IP address as the primary and mounts the same physical storage. The storage must be either a SAN LUN, NFS directories, or a local disk that has been mirrored over the network during normal operations. But, you can't have the same hostname and IP if the machine is running allowing the mirroring to take place. Thus, for a standby server, it must be powered off and take ownership of the storage when powered on. You _can_ do realtime mirroring to the standby while it's running, but then you have some really complex issues to deal with as far as hostname and IP assignments when the primary host dies and you have to take over the name and IP on the spare server. This can be done with a reboot and using alternate config files, and might actually work better in a virtual environment than with a physical machine as VM guests tend to boot faster than physical hosts due to things like long pauses caused by hardware BIOS routines. The key to all of the above is proper identification of primary host failure. The biggest problem with this setup is the two brains issue. There are a number of network scenarios that can cause your backup server or monitoring software to think the primary host is offline when it's really not. The secondary thus comes up, and now you have two hosts of the same name and IP address on the network. This situation can cause a number of serious problems IMO, the best way to do high availability is to use an active/active cluster of any number of nodes you see fit to meet your performance and reliability needs. All hosts are live all the time and share he load. When one goes down client performance may simply drops a bit, but that's about the extent of the downside. It's inherently more straight forward to setup than the previous scenario, especially if you're using NFS storage. In this case, you'd build two identical Dovecot servers and have each mount the same NFS mail directory. Read the list archives for ways to mitigate the index file issue. Timo wrote a new director specifically to meet this need. Two other options for the shared storage are a fiber channel or iSCSI SAN, or using DRBD to mirror disks (or logical devices--RAID) over the network. Both of these solutions require using a cluster filesystem such as GFS2. These can be quite a bit more difficult to setup and get working properly than the NFS method, especially for less experienced sysadmins. They can also be more difficult to troubleshoot, especially for sysadmins lacking sufficient knowledge or aptitude with regard to storage hardware and low level Linux device drivers. Hope this helps you a bit. You probably won't find a how to document that spoon feeds you the steps for an exact build/setup of this. If you choose the DRBD route you might be able to get Eric to write you up a step-by-step of how he did his two node DRBD Dovecot cluster. Maybe he's already written one. :) -- Stan
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Stan, On 11/1/10 7:30 PM, Stan Hoeppner s...@hardwarefreak.com wrote: 1. How many of you have a remote site hot backup Dovecot IMAP server? +1 2. How are you replicating mailbox data to the hot backup system? C. Other Netapp Fabric MetroCluster, active IMAP/POP3 nodes at both sites mounting storage over NFS, and active/standby hardware load balancers in front. Probably more than most folks can afford, but it's pretty bulletproof. -Brad
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Am 03.11.2010 07:17, schrieb Stan Hoeppner: Hope this helps you a bit. Thanks a lot, Stan, for this explanation ... ! Stefan
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Op 3-11-2010 7:17, Stan Hoeppner schreef: Stefan G. Weichinger put forth on 11/2/2010 1:15 PM: A bit off-topic, sorry ... I want to set up a hot backup dovecot in a VM, aside the physical server, so I am very interested in the best practise to do so ... There isn't one. If there was Timo would have pointed you to the wiki. Doing server fail over is inherently problematic for a large number of reasons. The easiest way to implement it is to literally turn on the backup server (power on) when the primary fails. The backup comes up with the same hostname and IP address as the primary and mounts the same physical storage. The storage must be either a SAN LUN, NFS directories, or a local disk that has been mirrored over the network during normal operations. But, you can't have the same hostname and IP if the machine is running allowing the mirroring to take place. Thus, for a standby server, it must be powered off and take ownership of the storage when powered on. You _can_ do realtime mirroring to the standby while it's running, but then you have some really complex issues to deal with as far as hostname and IP assignments when the primary host dies and you have to take over the name and IP on the spare server. This can be done with a reboot and using alternate config files, and might actually work better in a virtual environment than with a physical machine as VM guests tend to boot faster than physical hosts due to things like long pauses caused by hardware BIOS routines. The key to all of the above is proper identification of primary host failure. The biggest problem with this setup is the two brains issue. There are a number of network scenarios that can cause your backup server or monitoring software to think the primary host is offline when it's really not. The secondary thus comes up, and now you have two hosts of the same name and IP address on the network. This situation can cause a number of serious problems IMO, the best way to do high availability is to use an active/active cluster of any number of nodes you see fit to meet your performance and reliability needs. All hosts are live all the time and share he load. When one goes down client performance may simply drops a bit, but that's about the extent of the downside. It's inherently more straight forward to setup than the previous scenario, especially if you're using NFS storage. In this case, you'd build two identical Dovecot servers and have each mount the same NFS mail directory. Read the list archives for ways to mitigate the index file issue. Timo wrote a new director specifically to meet this need. Two other options for the shared storage are a fiber channel or iSCSI SAN, or using DRBD to mirror disks (or logical devices--RAID) over the network. Both of these solutions require using a cluster filesystem such as GFS2. These can be quite a bit more difficult to setup and get working properly than the NFS method, especially for less experienced sysadmins. They can also be more difficult to troubleshoot, especially for sysadmins lacking sufficient knowledge or aptitude with regard to storage hardware and low level Linux device drivers. Hope this helps you a bit. You probably won't find a how to document that spoon feeds you the steps for an exact build/setup of this. If you choose the DRBD route you might be able to get Eric to write you up a step-by-step of how he did his two node DRBD Dovecot cluster. Maybe he's already written one. :) Hello, i am working primarly with FreeBSD, and the latest release has a service called HAST. See it as a mirrored disk over the network. You can install both machines with dovecot, and use the hast disk as the data storage. With CARP in the mix, when the master machine fails, it starts dovecot on the slave. This way you have a failover without user interference. I did not test it, but i hope when time permits, i can try to start testing this setup. regards. Johan Hendriks
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Johan Hendriks put forth on 11/3/2010 3:32 AM: Hello, i am working primarly with FreeBSD, and the latest release has a service called HAST. See it as a mirrored disk over the network. You can install both machines with dovecot, and use the hast disk as the data storage. With CARP in the mix, when the master machine fails, it starts dovecot on the slave. This way you have a failover without user interference. 1. How do you automatically redirect clients to the IP address of the slave when the master goes down? Is this seamless? What is the duration of server down seen by clients? Seconds, minutes? 2. When you bring the master back up after repairing the cause of the failure, does it automatically and correctly resume mirroring of the HAST device so it obtains the new emails that were saved to the slave while it was offline? How do you then put the master back into service and make the slave offline again? -- Stan
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Stan Hoeppner wrote: 1. How do you automatically redirect clients to the IP address of the slave when the master goes down? Is this seamless? What is the duration of server down seen by clients? Seconds, minutes? Doesn't this work as a regular linuxHA setup with shared storage? The primary dies, the secondary reconfigures its interface to work as the primary, all done. AKA IP-takeover I believe. /Per Jessen, Zürich
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Stan Hoeppner wrote: IMO, the best way to do high availability is to use an active/active cluster of any number of nodes you see fit to meet your performance and reliability needs. All hosts are live all the time and share he load. When one goes down client performance may simply drops a bit, but that's about the extent of the downside. It's inherently more straight forward to setup than the previous scenario, especially if you're using NFS storage. In this case, you'd build two identical Dovecot servers and have each mount the same NFS mail directory. Read the list archives for ways to mitigate the index file issue. Timo wrote a new director specifically to meet this need. Two other options for the shared storage are a fiber channel or iSCSI SAN, or using DRBD to mirror disks (or logical devices--RAID) over the network. Both of these solutions require using a cluster filesystem such as GFS2. These can be quite a bit more difficult to setup and get working properly than the NFS method, especially for less experienced sysadmins. They can also be more difficult to troubleshoot, especially for sysadmins lacking sufficient knowledge or aptitude with regard to storage hardware and low level Linux device drivers. Stan, what do you think about GlusterFS or Ceph? I am looking for a high available AND scalable solution based on distributed/replicated storages (maildirs). I am not talking about a simple NFS export or something like this. A real-time replication managed by dovecot would surely be the best way to keep data in sync and balance clients. -- Sven
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Op 3-11-2010 12:16, Stan Hoeppner schreef: Johan Hendriks put forth on 11/3/2010 3:32 AM: Hello, i am working primarly with FreeBSD, and the latest release has a service called HAST. See it as a mirrored disk over the network. You can install both machines with dovecot, and use the hast disk as the data storage. With CARP in the mix, when the master machine fails, it starts dovecot on the slave. This way you have a failover without user interference. 1. How do you automatically redirect clients to the IP address of the slave when the master goes down? Is this seamless? What is the duration of server down seen by clients? Seconds, minutes? 2. When you bring the master back up after repairing the cause of the failure, does it automatically and correctly resume mirroring of the HAST device so it obtains the new emails that were saved to the slave while it was offline? How do you then put the master back into service and make the slave offline again? The servers work with an virtual ip. Carp does this, i use it for the firewalls on our location. Server 1 has ip adres 192.168.1.1, server 2 gets 192.168.1.2, and the virtual addres is 192.168.1.3 The clients connect to the virtual address 192.168.1.3, and contact the machine wich is master at that moment. I do not know how the rebuild goes with hast, if the master provider goes down, like i said, i need to try and test it. Maybe an question on the freebsd-fs mailing list will answer this. More about HAST http://wiki.freebsd.org/HAST More about Carp http://www.freebsd.org/doc/nl/books/handbook/carp.html regards, Johan Hendriks
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Johan Hendriks wrote: The servers work with an virtual ip. Carp does this, i use it for the firewalls on our location. Server 1 has ip adres 192.168.1.1, server 2 gets 192.168.1.2, and the virtual addres is 192.168.1.3 The clients connect to the virtual address 192.168.1.3, and contact the machine wich is master at that moment. Yes, this is a normal Linux-HA setup. /Per Jessen, Zürich
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Quoting Stan Hoeppner s...@hardwarefreak.com: Johan Hendriks put forth on 11/3/2010 3:32 AM: Hello, i am working primarly with FreeBSD, and the latest release has a service called HAST. See it as a mirrored disk over the network. This is similar to the DRBD solution. With CARP in the mix, when the master machine fails, it starts dovecot on the slave. This way you have a failover without user interference. This is similar to heartbeat, or RHCS, etc. 1. How do you automatically redirect clients to the IP address of the slave when the master goes down? Is this seamless? What is the duration of server down seen by clients? Seconds, minutes? Usually there is a floating IP that the clients used. Which ever server is active has this IP assigned (usually in addition to another IP used for management and such). The transition time depends on how the master goes down. If you do an administrative shutdown or transfer, it is usually just a fraction of a second for the change to take affect, and maybe a bit longer for the router/switch to get the new MAC address for the IP and route things correctly. If the primary crashes/dies, then it is usually several seconds before the secondary confirms the primary is in trouble, makes sure it is really down (STOMITH), and takes over the IP, mounts any needed filesystems, and starts any needed services... In this case, the arp/MAC issue isn't really a problem because the transition takes a longer time. 2. When you bring the master back up after repairing the cause of the failure, does it automatically and correctly resume mirroring of the HAST device so it obtains the new emails that were saved to the slave while it was offline? How do you then put the master back into service and make the slave offline again? DRBD does (or at least can, it is configurable). Sometimes you might just do role reversal (old primary becomes secondary, old secondary stays the primary). Other times you might have the original primary become primary again (say, if the original primary has better hardware, etc). So, these things really depend on the use case, and the failure case... And are usually configurable. :) I can give two personal examples. First I have a file server, which is active-passive cluster. Since the hardware is identical, when one fails, it is promoted to primary. When the dead one comes back, it stays as secondary. It is all automatic via RHCS and DRBD using ext3. Always feels like I'm wasting a machine, but it is rock solid... Second I have a mail cluster which is active-active (still RHCS but with DRBD+GFS2). When both nodes are up, one does the pop/imap, mailing list web/cli/email interface, and slave LDAP services, while the other node does the mailing list processing, SMTP processing, anti-virus/spam processing, etc. When one machine goes down, the services on that machine migrate automatically to the other machine. When the machine comes back up, the services migrate back to their home machine. Time for failover is a second or two for an admin failover, and for a crash/etc maybe 15-30 seconds max for the fileserver, and 10-15 seconds for the mail server. During the failover, connections may hang or fail, but most clients just retry the connection and get the new machine without user intervention (or in the case of email clients, sometimes they annoying ask for the password again, but that is not too bad). I've never had anyone contact me during either type of failover, which makes me think they either don't notice, or they write it off as a normal network hiccup kind of thing (well, they did contact me once, when the failover failed, and the service went completely down, but that was my fault). So, again, the answer is, as always, it depends... -- Stan -- Eric Rostetter The Department of Physics The University of Texas at Austin This message is provided AS IS without warranty of any kind, either expressed or implied. Use this message at your own risk.
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Quoting Johan Hendriks joh.hendr...@gmail.com: I do not know how the rebuild goes with hast, if the master provider goes down, like i said, i need to try and test it. Maybe an question on the freebsd-fs mailing list will answer this. More about HAST http://wiki.freebsd.org/HAST Sounds/looks a lot like DRBD. The config snip shown is about the same for both.. But I don't think it is DRBD, but rather based on DRBD... DRBD generally (configurable) will restart a rebuild as soon as a host comes up and contacts the other host(s), assuming it can. If it can't for any reason, then it of course sits and waits for manual intervention or for something to change so it can continue. -- Eric Rostetter The Department of Physics The University of Texas at Austin This message is provided AS IS without warranty of any kind, either expressed or implied. Use this message at your own risk.
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Hi, On 2010-11-02 10:30, Stan Hoeppner wrote: 1. How many of you have a remote site hot backup Dovecot IMAP server? Only local (same room, different box). Copying to a remote location in irregular intervals. 2. How are you replicating mailbox data to the hot backup system? A. rsync B. DRBD+GFS2 C. Other C. - rdiff-backup (versioned). Patrick. -- Key ID: 0x86E346D4http://patrick-nagel.net/key.asc Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4 signature.asc Description: OpenPGP digital signature
Re: [Dovecot] remote hot site, IMAP replication or cluster over WAN
Am 02.11.2010 03:30, schrieb Stan Hoeppner: Taking a survey. 1. How many of you have a remote site hot backup Dovecot IMAP server? 2. How are you replicating mailbox data to the hot backup system? A. rsync B. DRBD+GFS2 C. Other A bit off-topic, sorry ... I want to set up a hot backup dovecot in a VM, aside the physical server, so I am very interested in the best practise to do so ... Stefan
[Dovecot] remote hot site, IMAP replication or cluster over WAN
Taking a survey. 1. How many of you have a remote site hot backup Dovecot IMAP server? 2. How are you replicating mailbox data to the hot backup system? A. rsync B. DRBD+GFS2 C. Other Thanks. -- Stan
