Re: CARP not leaving backup state
William Stuart wrote: Hello everyone, I am sorry for not mentioning it was a vmWare instance. The packet replay seemed to be the culprit. This occured when we moved the image to a vmWare host running vmWare ESX 3.5 from 3.0. Our working theory is that under 3.5 pernicious mode works differently than under 3.0 and replays all of the traffic. Has anyone else experienced this problem? We have also experienced problems with CARP when moving ESX from 3.0i to 3.5. No solution yet.
Re: CARP not leaving backup state
We have also experienced problems with CARP when moving ESX from 3.0i to 3.5. No solution yet. Have tried to tweak the vSwitch settings ? I remember I made CARP work with 2 OpenBSD 4.2 VM on ESX 3.5 after changing some settings in the networking properties... (I know I should document what I do...) Denis
Re: CARP not leaving backup state
Hello everyone, I am sorry for not mentioning it was a vmWare instance. The packet replay seemed to be the culprit. This occured when we moved the image to a vmWare host running vmWare ESX 3.5 from 3.0. Our working theory is that under 3.5 pernicious mode works differently than under 3.0 and replays all of the traffic. Has anyone else experienced this problem? Stuart Henderson wrote: On 2008-07-19, William Stuart [EMAIL PROTECTED] wrote: Thanks everyone I figured it out! 19:13:46.334037 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 19:13:46.334299 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] Something is mirroring and replaying all the packets back. Grrr. Must be a vmWare config issue. Anyone asking about any wierd problems, _please_ mention any VMs that may be involved early on in the thread... as always, a dmesg would be a good starting point.
Re: CARP not leaving backup state
On 2008-07-21, William Stuart [EMAIL PROTECTED] wrote: I am sorry for not mentioning it was a vmWare instance. The packet replay seemed to be the culprit. This occured when we moved the image to a vmWare host running vmWare ESX 3.5 from 3.0. Our working theory is that under 3.5 pernicious mode works differently than under 3.0 and replays all of the traffic. Has anyone else experienced this problem? No vmware here, but I have noticed some emulator or other (maybe it was simh) returning transmitted ethernet frames on the receive side. Is it any better with vic(4)? (and can you check if basic functions work on ESX 3.x so the manual page, which only mentions 2.x, can be updated?)
Re: CARP not leaving backup state
On 2008-07-19, William Stuart [EMAIL PROTECTED] wrote: Thanks everyone I figured it out! 19:13:46.334037 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 19:13:46.334299 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] Something is mirroring and replaying all the packets back. Grrr. Must be a vmWare config issue. Anyone asking about any wierd problems, _please_ mention any VMs that may be involved early on in the thread... as always, a dmesg would be a good starting point.
Re: CARP not leaving backup state
Thank you for the idea. I did try that, but it still did not become MASTER. I had to give up on carp (and redundancy altogether) and run with aliases for now. Vinicius Vianna wrote: Hi William, I don't know for sure, but I remember dealing with this kind of problem and setting preempt did work, maybe worth a try: /etc/sysctl.conf: net.inet.carp.preempt=1 Anyone else? HTH, Vinicius William Stuart escreveu: (Sorry if this is a dupe, not sure if you had to be a subscriber to send to the list) Hello all, I am a new to OpenBSD but not *nix in general... I have two systems running OpenBSD 4.2. It has 9 carp interfaces, and has been running fine for months. All of a sudden, both systems are in BACKUP state. I halted one of the systems then on the remaining system rebooted, shut down and restarted, run ifconfig carp1 state master, changed the sysctls, removed the hostname files, rebooted, then replaced the hostname files, fiddled with the advskew and lots of other things. Even with no other system running, carp will not go into MASTER state, period, no errors, no logs. I tried setting net.inet.carp.log=1 and 2 and 1000, I see no logs anywhere in /var/log. Anything else I can look at? William
Re: CARP not leaving backup state
Hi Are you sure that all the interfaces you have configured carp on have link and can connect to each other? (I've seen similar behaviour caused by defective NICs: receive buffer not receiving while send buffer still sending - try ping on all interfaces) Is lo up? Is there any other router on the same network segment that propagates the same VHID with a better metric (i suppose that a VRRP router on the same net could cause trouble if it uses the same VHIDs). no answers, just questions, I know ... hth /markus
Re: CARP not leaving backup state
Markus Wernig wrote: Hi Are you sure that all the interfaces you have configured carp on have link and can connect to each other? (I've seen similar behaviour caused by defective NICs: receive buffer not receiving while send buffer still sending - try ping on all interfaces) Is lo up? Is there any other router on the same network segment that propagates the same VHID with a better metric (i suppose that a VRRP router on the same net could cause trouble if it uses the same VHIDs). no answers, just questions, I know ... hth /markus Don't mind the questions! Just hoping someone will ask one where I say, I am such an IDIOT! Why didn't I think of that! Yes, it had link (I was ssh'ing to the configured interface). They can connect to each other (I would assume if they couldn't see each other I would have a MASTER/MASTER issue and not a BACKUP/BACKUP issue), I ended up halting one of them to see if I can get just one to go MASTER when it was alone, no dice. I did a tcpdump and did not see any other VRRP traffic.
Re: CARP not leaving backup state
If you tcpdump do you see any carp traffic at all (ip proto 112)? Upon reboot? And you did enable carp preemption on both hosts (sysctl net.inet.carp.preempt=1)?
Re: CARP not leaving backup state
On Fri, Jul 18, 2008 at 01:26:07PM -0700, William Stuart wrote: Markus Wernig wrote: Hi Are you sure that all the interfaces you have configured carp on have link and can connect to each other? (I've seen similar behaviour caused by defective NICs: receive buffer not receiving while send buffer still sending - try ping on all interfaces) Is lo up? Is there any other router on the same network segment that propagates the same VHID with a better metric (i suppose that a VRRP router on the same net could cause trouble if it uses the same VHIDs). no answers, just questions, I know ... hth /markus Don't mind the questions! Just hoping someone will ask one where I say, I am such an IDIOT! Why didn't I think of that! Yes, it had link (I was ssh'ing to the configured interface). They can connect to each other (I would assume if they couldn't see each other I would have a MASTER/MASTER issue and not a BACKUP/BACKUP issue), I ended up halting one of them to see if I can get just one to go MASTER when it was alone, no dice. I did a tcpdump and did not see any other VRRP traffic. Is there any chance another CARP segment has recently been added to the same switch / stack? I've seen CARP vhid's leak through to other broadcast domains on some Avaya switches. -- Jason Dixon DixonGroup Consulting http://www.dixongroup.net/
Re: CARP not leaving backup state
Thanks everyone I figured it out! 19:13:46.334037 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] 19:13:46.334299 CARPv2-advertise 36: vhid=50 advbase=1 advskew=0 demote=0 (DF) [tos 0x10] Something is mirroring and replaying all the packets back. Grrr. Must be a vmWare config issue. William Jason Dixon wrote: If you'd like to send me your full `ifconfig -a` and pf.conf from both systems I'll take a look. -J. On Fri, Jul 18, 2008 at 06:19:22PM -0700, William Stuart wrote: Not before the failure, but I did test negative demoting as a fix.. -bash-3.2# ifconfig -g carp carp: carp demote count 0 Jason Dixon wrote: Any chance you've been adjusting your carpdemote counters? -J. On Fri, Jul 18, 2008 at 06:04:19PM -0700, William Stuart wrote: Gotcha. Nope, only one set of OpenBSD boxes. Just as a test, I replaced the carp config for one of the IP addresses and changed the VHID. still backup. -bash-3.2# ifconfig lo0: flags=8049UP,LOOPBACK,RUNNING,MULTICAST mtu 33168 groups: lo inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff00 em0: flags=8943UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST mtu 1500 lladdr 00:50:56:a0:4f:81 groups: egress media: Ethernet autoselect (1000baseT full-duplex,master) status: active inet6 fe80::250:56ff:fea0:4f81%em0 prefixlen 64 scopeid 0x1 inet 172.19.161.64 netmask 0x broadcast 172.19.161.64 enc0: flags=0 mtu 1536 carp5: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr 00:00:5e:00:01:32 carp: BACKUP carpdev em0 vhid 50 advbase 1 advskew 0 groups: carp inet6 fe80::200:5eff:fe00:132%carp5 prefixlen 64 scopeid 0x4 inet 172.19.161.67 netmask 0xff00 broadcast 172.19.161.255 -bash-3.2# cat /etc/hostname.carp5 inet 172.19.161.67 255.255.255.0 172.19.161.255 vhid 50 carpdev em0 pass blahblahblah Jason Dixon wrote: I didn't mean someone else had. I meant that perhaps you setup another pair on another network, but on the same switch. It happened to us, and even though we didn't see leaking carp/vrrp packets, the problem disappeared as soon as we changed the vhid to something unique. Oh well, try try again. -J. On Fri, Jul 18, 2008 at 03:58:56PM -0700, William Stuart wrote: Nope. Confirmed with tcpdump. No other carp/vrrp. Plus, where I am, no one else can spell B-S-D let alone install one, let alone setup carp. William Jason Dixon wrote: On Fri, Jul 18, 2008 at 01:26:07PM -0700, William Stuart wrote: Markus Wernig wrote: Hi Are you sure that all the interfaces you have configured carp on have link and can connect to each other? (I've seen similar behaviour caused by defective NICs: receive buffer not receiving while send buffer still sending - try ping on all interfaces) Is lo up? Is there any other router on the same network segment that propagates the same VHID with a better metric (i suppose that a VRRP router on the same net could cause trouble if it uses the same VHIDs). no answers, just questions, I know ... hth /markus Don't mind the questions! Just hoping someone will ask one where I say, I am such an IDIOT! Why didn't I think of that! Yes, it had link (I was ssh'ing to the configured interface). They can connect to each other (I would assume if they couldn't see each other I would have a MASTER/MASTER issue and not a BACKUP/BACKUP issue), I ended up halting one of them to see if I can get just one to go MASTER when it was alone, no dice. I did a tcpdump and did not see any other VRRP traffic. Is there any chance another CARP segment has recently been added to the same switch / stack? I've seen CARP vhid's leak through to other broadcast domains on some Avaya switches.
CARP not leaving backup state
(Sorry if this is a dupe, not sure if you had to be a subscriber to send to the list) Hello all, I am a new to OpenBSD but not *nix in general... I have two systems running OpenBSD 4.2. It has 9 carp interfaces, and has been running fine for months. All of a sudden, both systems are in BACKUP state. I halted one of the systems then on the remaining system rebooted, shut down and restarted, run ifconfig carp1 state master, changed the sysctls, removed the hostname files, rebooted, then replaced the hostname files, fiddled with the advskew and lots of other things. Even with no other system running, carp will not go into MASTER state, period, no errors, no logs. I tried setting net.inet.carp.log=1 and 2 and 1000, I see no logs anywhere in /var/log. Anything else I can look at? William
CARP not leaving backup state
Hello all, I am a new to OpenBSD but not *nix in general... I have two systems running OpenBSD 4.2. It has 9 carp interfaces, and has been running fine for months. All of a sudden, both systems are in BACKUP state. I have rebooted, shut down and restarted, run ifconfig carp1 state master, changed the sysctls, removed the hostname files, rebooted, then replaced the hostname files, run with only one system up, fiddled with the advskew and lots of other things. Even with no other system running, carp will not go into MASTER state, period, it won't try, it won't log. I tried setting net.inet.carp.log=1 and 2 and 1000, I see no logs anywhere in /var/log. Anything else I can look at? William
Re: CARP not leaving backup state
Hi William, I don't know for sure, but I remember dealing with this kind of problem and setting preempt did work, maybe worth a try: /etc/sysctl.conf: net.inet.carp.preempt=1 Anyone else? HTH, Vinicius William Stuart escreveu: (Sorry if this is a dupe, not sure if you had to be a subscriber to send to the list) Hello all, I am a new to OpenBSD but not *nix in general... I have two systems running OpenBSD 4.2. It has 9 carp interfaces, and has been running fine for months. All of a sudden, both systems are in BACKUP state. I halted one of the systems then on the remaining system rebooted, shut down and restarted, run ifconfig carp1 state master, changed the sysctls, removed the hostname files, rebooted, then replaced the hostname files, fiddled with the advskew and lots of other things. Even with no other system running, carp will not go into MASTER state, period, no errors, no logs. I tried setting net.inet.carp.log=1 and 2 and 1000, I see no logs anywhere in /var/log. Anything else I can look at? William