Alex Mathiasen created CLOUDSTACK-525:
-----------------------------------------
Summary: Cloudstack Redundant Virtual Router issues both MASTER.
Key: CLOUDSTACK-525
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-525
Project: CloudStack
Issue Type: Bug
Components: Install and Setup, KVM, Management Server, Network
Controller, Network Devices
Affects Versions: 4.0.0
Environment: Two servers with same hardware running Ubuntu 12.04 LTS.
Reporter: Alex Mathiasen
Priority: Critical
Steps:
- Two hosts added to cluster
- Add network offering with RVR set,
- Create instance using that network offering
- Look at the state of both virtual routers on each host being MASTER
Due to the state above, you will recieve errors with the following message:
More than one redundant virtual router is in MASTER state! Router r-15-VM and
router r-14-VM - Virtual router (name: r-15-VM, id: 15 and router (name:
r-14-VM, id: 15) are both in MASTER state! If the problem persist, restart both
of routers.
No errors in the logfiles, I have tried rebooting the virtual routers.
Migrating the second virtual router onto the same host will fix the state to
MASTER and BACKUP, but it's not the point - As the virtual router is supposed
to run on each host.
Looking at the console on both virtual routers, you will discover that
keepalived, conntrackd and haproxy failed to start.
Trying to ping the virtual routers by their local-link address. from
management-server, you will discover the following result:
This is the server running on same host as the instance.
root@cloudsrv1:/# ping 169.254.0.169
PING 169.254.0.169 (169.254.0.169) 56(84) bytes of data.
64 bytes from 169.254.0.169: icmp_req=1 ttl=64 time=0.193 ms
This is the server running on the second host, which should behave as BACKUP.
root@cloudsrv1:/mnt/mysql# ping 169.254.3.121
PING 169.254.3.121 (169.254.3.121) 56(84) bytes of data.
>From 169.254.0.1 icmp_seq=1 Destination Host Unreachable
View from management-server.log:
2012-11-21 09:19:42,066 DEBUG [agent.transport.Request]
(RouterStatusMonitor-1:null) Seq 3-2029663984: Sending { Cmd , MgmtId:
279278805451002, via: 3, Ver: v1, Flags: 100011,
[{"CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.0.169","router.name":"r-12-VM"},"wait":60}}]
}
2012-11-21 09:19:42,212 DEBUG [agent.transport.Request]
(AgentManager-Handler-8:null) Seq 3-2029663984: Processing: { Ans: , MgmtId:
279278805451002, via: 3, Ver: v1, Flags: 10,
[{"CheckRouterAnswer":{"state":"MASTER","isBumped":false,"result":true,"details":"Status:
MASTER&Bumped: NO","wait":0}}] }
2012-11-21 09:19:42,212 DEBUG [agent.transport.Request]
(RouterStatusMonitor-1:null) Seq 3-2029663984: Received: { Ans: , MgmtId:
279278805451002, via: 3, Ver: v1, Flags: 10, { CheckRouterAnswer } }
2012-11-21 09:19:42,213 DEBUG [agent.manager.AgentManagerImpl]
(RouterStatusMonitor-1:null) Details from executing class
com.cloud.agent.api.CheckRouterCommand: Status: MASTER&Bumped: NO
2012-11-21 09:19:42,267 DEBUG [agent.transport.Request]
(RouterStatusMonitor-1:null) Seq 5-1237057653: Sending { Cmd , MgmtId:
279278805451002, via: 5, Ver: v1, Flags: 100011,
[{"CheckRouterCommand":{"accessDetails":{"router.ip":"169.254.3.121","router.name":"r-13-VM"},"wait":60}}]
}
2012-11-21 09:19:42,421 DEBUG [agent.transport.Request]
(AgentManager-Handler-14:null) Seq 5-1237057653: Processing: { Ans: , MgmtId:
279278805451002, via: 5, Ver: v1, Flags: 10,
[{"CheckRouterAnswer":{"state":"MASTER","isBumped":false,"result":true,"details":"Status:
MASTER&Bumped: NO","wait":0}}] }
2012-11-21 09:19:42,421 DEBUG [agent.transport.Request]
(RouterStatusMonitor-1:null) Seq 5-1237057653: Received: { Ans: , MgmtId:
279278805451002, via: 5, Ver: v1, Flags: 10, { CheckRouterAnswer } }
2012-11-21 09:19:42,421 DEBUG [agent.manager.AgentManagerImpl]
(RouterStatusMonitor-1:null) Details from executing class
com.cloud.agent.api.CheckRouterCommand: Status: MASTER&Bumped: NO
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
