Re: what all touches the carp demote counter?

[Paul B. Henson]

On Fri, Oct 14, 2016 at 01:27:42PM -0700, Paul B. Henson wrote:
> Arg, I'm still having issues with the carp demote counter. I disabled
> ospfd for now, but something is still changing it. After a reboot
> without ospfd, the counter is changing between 0 and 1:

Ah, I tracked it down. I had configured another carp interface on the
new system which didn't yet have a corresponding interface on the old
system. I have the carp interfaces configured with explicit peer
addresses rather than using multicast, and evidentally the inability to
send a packet to the peer was causing the other carp interface to
twiddle the global carp demote counter, which popped up once I cranked
up the carp log level:

Oct 14 15:21:48 lisa /bsd: carp: carp1 demoted group carp by -1 to 2 (< 
snderrors)
Oct 14 15:21:52 lisa /bsd: carp1: ip_output failed: 64
Oct 14 15:21:54 lisa /bsd: carp: carp1 demoted group carp by 1 to 3 (> 
snderrors)
Oct 14 15:21:55 lisa /bsd: carp1: ip_output failed: 64
Oct 14 15:22:14 lisa /bsd: carp: carp1 demoted group carp by -1 to 2 (< 
snderrors)
Oct 14 15:22:18 lisa /bsd: carp1: ip_output failed: 64
Oct 14 15:22:20 lisa /bsd: carp: carp1 demoted group carp by 1 to 3 (> 
snderrors)

It doesn't do this if I remove the carppeer and use the default multicast;
that's an unexpected side effect of configuring a carppeer that might be
worth documenting. A down carppeer on one interface can impact the
functionality of all carp interfaces on the system.

Re: what all touches the carp demote counter?

[Paul B. Henson]

Arg, I'm still having issues with the carp demote counter. I disabled
ospfd for now, but something is still changing it. After a reboot
without ospfd, the counter is changing between 0 and 1:

bash-4.3# ifconfig -g carp
carp: carp demote count 1

bash-4.3# ifconfig -g carp
carp: carp demote count 0

bash-4.3# ifconfig -g carp
carp: carp demote count 1

bash-4.3# ifconfig -g carp
carp: carp demote count 0

And the carp interface is flapping:

Oct 14 13:17:17 lisa /bsd: carp0: state transition: BACKUP -> MASTER
Oct 14 13:17:23 lisa /bsd: carp0: state transition: MASTER -> BACKUP
Oct 14 13:17:43 lisa /bsd: carp0: state transition: BACKUP -> MASTER
Oct 14 13:17:49 lisa /bsd: carp0: state transition: MASTER -> BACKUP
Oct 14 13:18:08 lisa /bsd: carp0: state transition: BACKUP -> MASTER

There's not too much running; smtpd, sshd, npppd, dhcpd. Any suggestions
as to what might be screwing with the carp demote value?

Thanks...


root 1  0.0  0.0   440   520 ??  Is 1:14PM0:01.01 /sbin/init
root 21696  0.0  0.0  1044  1296 ??  Isp1:14PM0:00.00 syslogd: 
[priv] (syslogd)
_syslogd 22103  0.0  0.0  1044  1388 ??  Sp 1:14PM0:00.07 
/usr/sbin/syslogd
_pflogd   5335  0.0  0.0   684   400 ??  Sp 1:14PM0:00.02 pflogd: 
[running] -s 160 -i pfl
root 27252  0.0  0.0   620   600 ??  Is 1:14PM0:00.00 pflogd: 
[priv] (pflogd)
_ntp 16170  0.0  0.0   636  1472 ??  Isp1:14PM0:00.02 ntpd: dns 
engine (ntpd)
_ntp 15754  0.0  0.0   688  1540 ??  S I'm setting up a second router that's going to sit next to an existing
> one and become a redundant failover system. The current one is in
> production, and I've been converting some of the existing LAN subnets on it
> to use carp interfaces and making them primary and the new box
> secondary. I also set up a carp interface on the WAN side and made the
> new box primary for testing as that didn't exist before. That all
> worked fine when I set it up by hand, but when I rebooted the new box,
> the old box stayed primary for everything including the WAN interface,
> which I tracked down to the carp demote counter, which ended up at 2 on
> the new box after the reboot:
> 
> bash-4.3# ifconfig -g carp
> carp: carp demote count 2
> 
> After I manually decreased the demote counter by 2 back to 0 the WAN
> interface master switched back to the new box.
> 
> I'm not sure what's doing that at boot? I am running ospfd on the box,
> but I don't have any demote statements in my configuration. I'm also
> running npppd, but I don't see anything about that and carp demotion.
> What else might be setting carp demotion values?
> 
> Thanks...

Re: what all touches the carp demote counter?

[Paul B. Henson]

On Wed, Oct 12, 2016 at 08:37:59AM +0200, mxb wrote:

> But as R0me0 stated, you should probably re-check your configuration.

The configuration checked out. I rebooted a few more times, and I
couldn't reproduce the problem. I still have no idea why the carp
demotion counter was set to 2 the first time I rebooted. It doesn't seem
to be doing it anymore though. Thanks for all the suggestions though, it
helped to verify everything was set up right.

Re: what all touches the carp demote counter?

[mxb]

> On 11 okt. 2016, at 23:56, Paul B. Henson  wrote:
>
> Does pfsync fiddle with the carp
> demotion value even if it's not configured?


No.
But as R0me0 stated, you should probably re-check your configuration.

carp.preempt=1  (  /etc/sysctl.conf ) on both nodes, if not
node which toke over master roll will stay master until it goes down.

All default recommendations/“best practice” are in man pages.

//mxb

Re: what all touches the carp demote counter?

[Paul B. Henson]

On Tue, Oct 11, 2016 at 08:44:05AM +0200, mxb wrote:

> Master-Backup setup with pfsync in place, means that you synchronize
> states between boxes.  Then Master is rebooted, it becomes out-of-sync
> then it comes to states.  So until it is in sync with Backup (which
> became Master after reboot), it will not become Master.
> 
> This process is auto. Just need to wait.

I haven't set up pfsync yet, I need to upgrade the old box first. Right
now I'm just working with carp. Does pfsync fiddle with the carp
demotion value even if it's not configured?

Thanks...

Re: what all touches the carp demote counter?

[mxb]

Master-Backup setup with pfsync in place, means that you synchronize states
between boxes.
Then Master is rebooted, it becomes out-of-sync then it comes to states.
So until it is in sync with Backup (which became Master after reboot), it will
not become Master.

This process is auto. Just need to wait.

//mxb

> On 11 okt. 2016, at 03:58, Paul B. Henson  wrote:
>
> On Mon, Oct 10, 2016 at 09:43:56PM -0300, R0me0 *** wrote:
>
>> Did you adjust advskew value on the machine you want to be Backup ?
>
> Yes, the backup has an advskew of 5 and the primary an advskew of 1. As
> I mentioned, when I first configured the interfaces by hand the two
> systems properly negotiated master/backup roles, it was only after I
> rebooted the one that was supposed to be primary on this interface that
> it came up as backup, and I traced it to the fact the the carp demote value
> was set to 2. When I manually changed the carp demote value to 0, the
> system once again pre-empted the master role on the interface.
>
> I'm just not sure what is twiddling with the carp demotion value. Unless
> ospdf does it by default? The man page for the config file reads like it
> would only do it if you explicitly include the demote keyword in the
> area or interface section.
>
> Thanks for the suggestion though.

Re: what all touches the carp demote counter?

[R0me0 ***]

Hello sorry my bad english

So, Let's debug

Review carp/pfsync ( NODE1-carp0/NODE2-carp0 samepassword  and same vhid
 for each pair ) ( pfsync syncdev ) ( /etc/hostname.pfsync0 = up syndev
IFACE )

check default gateway on both ( /etc/mygate)  /  sysctl ip.forwarding=1 and
carp.preempt=1  (  /etc/sysctl.conf )

( pf rules )

put carp and pfsync rules on the TOP of your rules ( for debug purpose set
skip on { lo0 $pfsyncdev }

=> pass quick on { $carpdev $carpdev2 $carpdev3 } proto carp keep state
(no-sync) <=

- Check with tcpdump pflog if carp packets are being dropped

- Check if all carps interfaces are as  MASTER in the current node and
check if all is as BACKUP on another

- Check if in the current backup node the states are syncing ( systat
states )  ( compare on both pfctl -ss | wc -l )  almost the same quantity .


Bring up to MASTER  the primary node ( ifconfig -g carp carpdemote 30 on
current master node ) the slave need to have a lower value of carpdemote
 and  NOT DEFINE advskew in the Primary leave default ( 0 )  just on backup
set  advskew and put a high value ( advskew 100 )  ( hostname.carp )

- Check  carpdemote on the new primary master  ( ifconfig -g carp )  if the
value is not 0 set to

- Reboot the slave node ... and when back check if  keep as slave ( check
systat states ) ( compare on both pfctl -ss | wc -l ) must be almost equal.

- Check carpdemote on slave should be 0

Reboot the Primary/Master and when back its supposed to be Master


* if you have a huge traffic ( the node you rebooted must delay until
states be syncronized )

About ospf I have no experience working with carp.

one more time sorry any typo

[]'s



2016-10-10 22:58 GMT-03:00 Paul B. Henson :

> On Mon, Oct 10, 2016 at 09:43:56PM -0300, R0me0 *** wrote:
>
> > Did you adjust advskew value on the machine you want to be Backup ?
>
> Yes, the backup has an advskew of 5 and the primary an advskew of 1. As
> I mentioned, when I first configured the interfaces by hand the two
> systems properly negotiated master/backup roles, it was only after I
> rebooted the one that was supposed to be primary on this interface that
> it came up as backup, and I traced it to the fact the the carp demote value
> was set to 2. When I manually changed the carp demote value to 0, the
> system once again pre-empted the master role on the interface.
>
> I'm just not sure what is twiddling with the carp demotion value. Unless
> ospdf does it by default? The man page for the config file reads like it
> would only do it if you explicitly include the demote keyword in the
> area or interface section.
>
> Thanks for the suggestion though.

Re: what all touches the carp demote counter?

[Paul B. Henson]

On Mon, Oct 10, 2016 at 09:43:56PM -0300, R0me0 *** wrote:

> Did you adjust advskew value on the machine you want to be Backup ?

Yes, the backup has an advskew of 5 and the primary an advskew of 1. As
I mentioned, when I first configured the interfaces by hand the two
systems properly negotiated master/backup roles, it was only after I
rebooted the one that was supposed to be primary on this interface that
it came up as backup, and I traced it to the fact the the carp demote value
was set to 2. When I manually changed the carp demote value to 0, the
system once again pre-empted the master role on the interface.

I'm just not sure what is twiddling with the carp demotion value. Unless
ospdf does it by default? The man page for the config file reads like it
would only do it if you explicitly include the demote keyword in the
area or interface section.

Thanks for the suggestion though.

Re: what all touches the carp demote counter?

[R0me0 ***]

Hello Paul,

Did you adjust advskew value on the machine you want to be Backup ?

For example:


Primary/Master

# cat hostname.carp0

vhid 1 cardev em0 pass THEPASSWORD
inet 10.20.30.40 255.255.255.0


Slave/Backup
# cat hostname.carp0

vhid 1 cardev em0 advskew 100 pass THEPASSWORD
inet 10.20.30.40 255.255.255.0

I think could be it

Regards,

2016-10-10 20:30 GMT-03:00 Paul B. Henson :

> I'm setting up a second router that's going to sit next to an existing
> one and become a redundant failover system. The current one is in
> production, and I've been converting some of the existing LAN subnets on it
> to use carp interfaces and making them primary and the new box
> secondary. I also set up a carp interface on the WAN side and made the
> new box primary for testing as that didn't exist before. That all
> worked fine when I set it up by hand, but when I rebooted the new box,
> the old box stayed primary for everything including the WAN interface,
> which I tracked down to the carp demote counter, which ended up at 2 on
> the new box after the reboot:
>
> bash-4.3# ifconfig -g carp
> carp: carp demote count 2
>
> After I manually decreased the demote counter by 2 back to 0 the WAN
> interface master switched back to the new box.
>
> I'm not sure what's doing that at boot? I am running ospfd on the box,
> but I don't have any demote statements in my configuration. I'm also
> running npppd, but I don't see anything about that and carp demotion.
> What else might be setting carp demotion values?
>
> Thanks...