Hi,
Yes, the aim is to get complete functionality failover(/back) between
node 1 and 3 to provide media services, admin page, user portals and all
the VM data. Additionally to that, all the relevant data should be
mirrored on both sites. It's ambitious I know.
Yes, postgres DB is replicated already. I'm using DRBD to mirror it
between node 1 and node 2. See details below.
The system -hamon(stands for HA MONitor)- treats the sipXecs as much as
a black-box as possible. It manages service IP, file replication and
services(sipXecs, named, vsftpd, etc). At the moment it's around 3100
lines of code including an installer script, which can setup all 3 nodes
in one go.
The service IP management works like this: the 3 nodes have 4 IP
addresses. Node 1 has a "private IP" which it comes up with, then checks
if the service IP is already occupied by node 3. If not, it switches to
service IP. Node 2 has a constant IP. Node 3 comes up with its "private
IP" again and as soon as it sees, that the service IP is not occupied by
node 1, it takes over it. There's a mechanism which regularly checks the
availability of network media(cable connected), availability of external
IP(is the network separated) and whether there's another node with
service IP to avoid situations when accidentally both node 1 and node 3
came up with service IP.
The file replication is a bit more complex. The idea behind is, that all
the necessary data is copied between node 1 and node 2. When node 3
comes up, it mounts the replicated data over NFS from node 2 and runs
using that. Remember, that node 2 and 3 are on the same site. The
connection between node 2 and 3 is expected to be permanent, which can
be achieved by a bonded network IF I think(not tested yet).
The data mirroring works the following way: the postgres DB is
replicated between node 1 and node 2 using DRBD and the less sensitive
files are replicated using rsync/lsync. When node 1 is active, it uses
lsyncd to push changed files to node 2 and when it's inactive -while
waiting for the service IP to be unoccupied-, it pulls data using rsync
from node 2. Node 2 just exports the DRBD mirror and the rsynced dirs to
node 3 using NFS. Node 3 when active mounts the exports from node 2.
When a node has the service IP _and_ the data mirroring is set up
properly, it starts sipXecs service and performs health checks
periodically(network, shares, SIP trunk) and if a test fails, it will
headshot sipXecs, release service IP to allow other node to take over as
soon as possible. Then story starts again by waiting for service IP to
become unoccupied.
This is it in a nutshell. Once the hamon is in a beta testable state and
if there are volunteers, I can send out the code for you guys to have a
look and get feedback. Once it's done and there's an interest for it, we
can share final code with the community.
Please let me know about your thoughts.
Thanks,
Chris
PS: The list of currently replicated directories is below.
DRBD replicated:
/var/lib/pgsql
rsync/lsyncd replicated:
/etc/sipxpbx
/etc/ssh
/root/.ssh
/var/named
/var/sipxdata/audiocodes
/var/sipxdata/certdb
/var/sipxdata/configserver
/var/sipxdata/mediaserver
/var/sipxdata/mrtg
/var/sipxdata/parkserver
/var/sipxdata/process-cfgver
/var/sipxdata/process-state
/var/sipxdata/reports
/var/sipxdata/sipdb
/var/sipxdata/sipxpage
/var/sipxdata/sipxpresence
/var/sipxdata/upgrade
-----Original Message-----
From: Douglas Hubler [mailto:[email protected]]
Sent: Wednesday, September 22, 2010 5:29 PM
To: sipXecs developer discussions
Subject: Re: [sipx-dev] 3 node HA
For those of us that do not recall the plan details, if you're looking
for hotswap on sipxconfig, there's quite a bit to mirror to get right
and honestly, no real need because it's not in critical path. Are you
looking to have hotswap of user portal portion of sipXconfig? That
makes sense, If so those files are not nec., but all of postgres is,
you have that setup already?
On Wed, Sep 22, 2010 at 11:36 AM, Krisztian Ganyai
<[email protected]> wrote:
> Hi,
>
> You might remember, that some months ago I was drafting an HA plan
with
> 3 nodes trying to solve the sipxconfig and mediaservices SPOF.
>
> The basic idea was to set up node 1 on site A and node 2 + 3 on site
B.
> Node 1(config+media services) and node 2(redundant proxy) form a
> traditional HA SIP cluster. When node 1 fails, the node 3 on site B
> comes up and takes over IP address and role of node 1 and vice versa.
> For this, some file/DB replication has to be in place and additionally
> some mechanism to make sure the system is always in a valid state. The
> system is supposed to stay fully functional with any 2 of the nodes
> running at a given time(excluding failover time).
>
> In the recent weeks I had some time to work on this and managed to put
> together the file replication and IP failover part, which is in a
quite
> good shape now. Last week I started adding the control of the sipXecs
> "on top of it" and there is some progress as well. When creating a
user
> on node 1 and pulling network cable out of it, shortly after the node
3
> comes up and the newly created user is visible there.
>
> Where I'm kind of stuck at the moment is the /var/sipxdata directory.
> Could someone please tell me which directories are _NOT_ necessary in
it
> from replication point of view? I suppose /var/sipxdata/tmp/ is not
and
> I'm not entirely sure about /var/sipxdata/process-state/. Are there
any
> others, which could be excluded?
>
> Thanks in advance,
> Chris
>
> PS: In case of questions, please don't hesitate to ask.
> _______________________________________________
> sipx-dev mailing list
> [email protected]
> List Archive: http://list.sipfoundry.org/archive/sipx-dev/
>
_______________________________________________
sipx-dev mailing list
[email protected]
List Archive: http://list.sipfoundry.org/archive/sipx-dev/
