Re: CARP and VRRP compliance

[Samuel Jayden]

Hello Theo,

It's disheartening to see the disparity in treatment between entities like
OpenBSD and larger corporations within these governance structures.
However, your resolve in the face of such challenges is commendable. The
creation of CARP, under the circumstances you described, not only serves as
a practical solution but also as a principled stand against the
monopolization of technology standards. This unwavering commitment is the
reason OpenBSD is so deeply respected and cherished.

Thank you for your perseverance and for setting an example of integrity in
the technology community.
This is why we love OpenBSD so much.

Kind regards
Sam

On Wed, Feb 14, 2024 at 7:26 PM Theo de Raadt  wrote:

> Stuart Henderson  wrote:
>
> > On 2024-02-13, Samuel Jayden  wrote:
> > > From the information provided in the link, it appears that CARP and
> VRRP
> > > protocols aren't inherently interoperable.
> >
> > They are different protocols - they *had* to be different because VRRP
> > was subject to patents. And if carp was changed now, it wouldn't be
> > interoperable with existing carp installations.
> >
> > > While Cisco may have attempted to address this by introducing a command
> > > like "disable-loop-detection carp" in its Nexus 1000V virtual router
> > > product, this solution unfortunately doesn't extend to standard router
> > > hardware, rendering it ineffective in many scenarios.
> >
> > That's not about interop beteeen carp and vrrp speakers, it's about
> > using carp (or vrrp or hsrp or similar) on a port attached to the
> > 'virtual switch'. See 'Information About Redundant Routing Protocols' on
> >
> https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_5_1/layer_2_switching/configuration/guide/n1000v_l2/n1000v_l2_7redundantroutingprot.html
> >
> > > Is it feasible to achieve CARP and VRRP interoperability through a
> > > user-space application?
> >
> > No. They are different protocols. For what you want to do, running VRRP
> > on the OpenBSD box might make some sense though. There are various
> > existing userland implementations of VRRP that might be able to run
> > on OpenBSD, probably with some work to port them - e.g. freevrrpd,
> > frr-vrrpd, vrrpd. Nothing already in the ports tree (if someone wanted
> > to try I'd suggest starting by looking at freevrrpd).
>
> This was my experience:
>
> VRRP was the first patent-encumbered protocol squeezed through the IETF
> process.
>
> The backers of that change in process were employees and laywers at a few
> major companies, but also tightly integrated into the IETF approval
> process.
>
> When we objected to the VRRP situation, they circled the wagons, not just
> to defend the VRRP patent, but to protect a future of patent's being OK in
> IETF processes.
>
> In response, OpenBSD carefully developed a similar mechanism called CARP,
> and the acronymn actually expands to "Cisco Asshole Redundancy Protocol",
> because the main traitors inside IETF were Cisco employees.
>
> Then we asked IETF for numbers to make this a unique protocol.  Unlike
> a recent threads where Tatu asked IETF for port 22 and they just gave it
> to him, the various number authorities inside IETF demanded that we follow
> the most stringent procedures for CARP.  Even to this day, IETF provides
> the various prototol numbers to some large corporate industry members
> without
> forcing them down those stringent procedures.
>
> As a result, we simply squatted on the VRRP numbers.  We gave them plenty
> of warning we would be doing this.  Over the following years, we heard some
> real anger IETF decision makers internally, but none of them re-visited our
> request for seperate numbers.  We never got numbers.  So CARP will stay
> where it is.
>
> One major bug was in VRRP on some HP product was found in the first year.
> CARP packets were incorrectly parsed as VRRP packets.  I don't remember
> the details, but I think it rebooted that HP device, probably a switch.
>
> Oh well.
>
>

Re: CARP and VRRP compliance

[Samuel Jayden]

Greetings,

I have now attained a deeper understanding of the topic at hand; thank you
for your insights. It appears that my requirements necessitate
communication between a Cisco router and VRRP, rather than CARP. Upon
reviewing the open-source projects you've recommended, here are my findings:

The vrrpd project seems quite distant from being readily compilable. It
exhibits a classic Linux developer's perspective, showing no inclination
towards ensuring compatibility with operating systems outside the Linux
realm.

I am still engaged with frr-vrrpd, yet, to my dismay, I haven't managed to
compile it thus far.

With freevrrpd, I am tantalizingly closer to a resolution. By crafting
minor patches, I've successfully compiled it, albeit necessitating the
deactivation of netgraph code.

Upon conducting a VRRP test between OpenBSD + freevrrpd and a Cisco Router,
I observed that both devices persisted in identifying themselves as the
master. Monitoring the relevant interface with tcpdump allowed me to
perceive packets emanating from the Cisco Router; however, there was a
conspicuous absence of VRRP packets from the OpenBSD system. It seems
plausible that disabling the netgraph code contributed to this predicament.

Should there exist an equivalent to netgraph within OpenBSD, I am eager to
explore that avenue.

Thanks.
Sam


On Wed, Feb 14, 2024 at 2:06 PM Stuart Henderson 
wrote:

> On 2024-02-13, Samuel Jayden  wrote:
> > From the information provided in the link, it appears that CARP and VRRP
> > protocols aren't inherently interoperable.
>
> They are different protocols - they *had* to be different because VRRP
> was subject to patents. And if carp was changed now, it wouldn't be
> interoperable with existing carp installations.
>
> > While Cisco may have attempted to address this by introducing a command
> > like "disable-loop-detection carp" in its Nexus 1000V virtual router
> > product, this solution unfortunately doesn't extend to standard router
> > hardware, rendering it ineffective in many scenarios.
>
> That's not about interop beteeen carp and vrrp speakers, it's about
> using carp (or vrrp or hsrp or similar) on a port attached to the
> 'virtual switch'. See 'Information About Redundant Routing Protocols' on
>
> https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_5_1/layer_2_switching/configuration/guide/n1000v_l2/n1000v_l2_7redundantroutingprot.html
>
> > Is it feasible to achieve CARP and VRRP interoperability through a
> > user-space application?
>
> No. They are different protocols. For what you want to do, running VRRP
> on the OpenBSD box might make some sense though. There are various
> existing userland implementations of VRRP that might be able to run
> on OpenBSD, probably with some work to port them - e.g. freevrrpd,
> frr-vrrpd, vrrpd. Nothing already in the ports tree (if someone wanted
> to try I'd suggest starting by looking at freevrrpd).
>
> --
> Please keep replies on the mailing list.
>
>

Re: CARP and VRRP compliance

[Theo de Raadt]

Stuart Henderson  wrote:

> On 2024-02-13, Samuel Jayden  wrote:
> > From the information provided in the link, it appears that CARP and VRRP
> > protocols aren't inherently interoperable.
> 
> They are different protocols - they *had* to be different because VRRP
> was subject to patents. And if carp was changed now, it wouldn't be
> interoperable with existing carp installations.
> 
> > While Cisco may have attempted to address this by introducing a command
> > like "disable-loop-detection carp" in its Nexus 1000V virtual router
> > product, this solution unfortunately doesn't extend to standard router
> > hardware, rendering it ineffective in many scenarios.
> 
> That's not about interop beteeen carp and vrrp speakers, it's about
> using carp (or vrrp or hsrp or similar) on a port attached to the
> 'virtual switch'. See 'Information About Redundant Routing Protocols' on
> https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_5_1/layer_2_switching/configuration/guide/n1000v_l2/n1000v_l2_7redundantroutingprot.html
> 
> > Is it feasible to achieve CARP and VRRP interoperability through a
> > user-space application?
> 
> No. They are different protocols. For what you want to do, running VRRP
> on the OpenBSD box might make some sense though. There are various
> existing userland implementations of VRRP that might be able to run
> on OpenBSD, probably with some work to port them - e.g. freevrrpd,
> frr-vrrpd, vrrpd. Nothing already in the ports tree (if someone wanted
> to try I'd suggest starting by looking at freevrrpd).

This was my experience:

VRRP was the first patent-encumbered protocol squeezed through the IETF process.

The backers of that change in process were employees and laywers at a few
major companies, but also tightly integrated into the IETF approval process.

When we objected to the VRRP situation, they circled the wagons, not just
to defend the VRRP patent, but to protect a future of patent's being OK in
IETF processes.

In response, OpenBSD carefully developed a similar mechanism called CARP,
and the acronymn actually expands to "Cisco Asshole Redundancy Protocol",
because the main traitors inside IETF were Cisco employees.

Then we asked IETF for numbers to make this a unique protocol.  Unlike
a recent threads where Tatu asked IETF for port 22 and they just gave it
to him, the various number authorities inside IETF demanded that we follow
the most stringent procedures for CARP.  Even to this day, IETF provides
the various prototol numbers to some large corporate industry members without
forcing them down those stringent procedures.

As a result, we simply squatted on the VRRP numbers.  We gave them plenty
of warning we would be doing this.  Over the following years, we heard some
real anger IETF decision makers internally, but none of them re-visited our
request for seperate numbers.  We never got numbers.  So CARP will stay
where it is.

One major bug was in VRRP on some HP product was found in the first year.
CARP packets were incorrectly parsed as VRRP packets.  I don't remember
the details, but I think it rebooted that HP device, probably a switch.

Oh well.

Re: CARP and VRRP compliance

[Stuart Henderson]

On 2024-02-13, Samuel Jayden  wrote:
> From the information provided in the link, it appears that CARP and VRRP
> protocols aren't inherently interoperable.

They are different protocols - they *had* to be different because VRRP
was subject to patents. And if carp was changed now, it wouldn't be
interoperable with existing carp installations.

> While Cisco may have attempted to address this by introducing a command
> like "disable-loop-detection carp" in its Nexus 1000V virtual router
> product, this solution unfortunately doesn't extend to standard router
> hardware, rendering it ineffective in many scenarios.

That's not about interop beteeen carp and vrrp speakers, it's about
using carp (or vrrp or hsrp or similar) on a port attached to the
'virtual switch'. See 'Information About Redundant Routing Protocols' on
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_5_1/layer_2_switching/configuration/guide/n1000v_l2/n1000v_l2_7redundantroutingprot.html

> Is it feasible to achieve CARP and VRRP interoperability through a
> user-space application?

No. They are different protocols. For what you want to do, running VRRP
on the OpenBSD box might make some sense though. There are various
existing userland implementations of VRRP that might be able to run
on OpenBSD, probably with some work to port them - e.g. freevrrpd,
frr-vrrpd, vrrpd. Nothing already in the ports tree (if someone wanted
to try I'd suggest starting by looking at freevrrpd).

-- 
Please keep replies on the mailing list.

Re: CARP and VRRP compliance

[Philipp Buehler]

Am 13.02.2024 19:07 schrieb Samuel Jayden:

Also I've another question:
Is it feasible to achieve CARP and VRRP interoperability through a
user-space application?


One step back.. you're looking for using one cisco router and one
OpenBSD box as a redundant pair? I've no idea and in over 20y I did
not consider doing this.

If you think about how an OpenBSD pair (failover/load between 
themselves)
and "on the other side" a Cisco pair using VRRP (acting betweeen 
themselves),
I can tell that this works w/o having a stamped letter with some crayon 
on it.



--
pb

Re: CARP and VRRP compliance

[Samuel Jayden]

Hello Marcus,

Thank you for your response.

>From the information provided in the link, it appears that CARP and VRRP
protocols aren't inherently interoperable.
While Cisco may have attempted to address this by introducing a command
like "disable-loop-detection carp" in its Nexus 1000V virtual router
product, this solution unfortunately doesn't extend to standard router
hardware, rendering it ineffective in many scenarios.

Also I've another question:
Is it feasible to achieve CARP and VRRP interoperability through a
user-space application?
I am curious if there are any existing solutions or approaches that
leverage user-space applications to bridge the interoperability gap between
CARP and VRRP.
If anyone has insights or experiences in this area, I would greatly
appreciate hearing about them.

Thank you for considering my inquiries.

Best regards
Sam

On Tue, Feb 13, 2024 at 8:26 PM Marcus MERIGHI  wrote:

> Hello Samuel,
>
> samueljaydan1...@gmail.com (Samuel Jayden), 2024.02.13 (Tue) 17:35 (CET):
> > I am reaching out to seek guidance on creating redundancy between a Cisco
> > Router and OpenBSD. After conducting extensive research on the subject, I
> > find myself in need of clarification on a specific point.
>
> This has some background info for you:
>
> https://mwl.io/archives/1866
>
> Marcus
>

Re: CARP and VRRP compliance

[Marcus MERIGHI]

Hello Samuel, 

samueljaydan1...@gmail.com (Samuel Jayden), 2024.02.13 (Tue) 17:35 (CET):
> I am reaching out to seek guidance on creating redundancy between a Cisco
> Router and OpenBSD. After conducting extensive research on the subject, I
> find myself in need of clarification on a specific point.

This has some background info for you:

https://mwl.io/archives/1866

Marcus

Re: carp flapping

[Nick Holland]

Followup...

On 5/12/23 08:17, Stuart Henderson wrote:

On 2023-05-12, Nick Holland  wrote:

...

I had several other people suggest network problems.  I'm not going to
say "impossible" or even "unlikely", but my understanding is that the
two machines are both plugged into the same switch, in the same rack.




I've since had someone more familiar with the physical environment say
my blind trust in their switch hw may be slightly misplaced. :)


You can also look at

netstat -ni -I ixl0
netstat -ni -I ixl0 -e
kstat ixl0:::



These looked REALLY clean.  no drops, fails or collisions.


which may give some other clues

even pfctl -si might have something relevant


Several people pointed out I was using the default advskew of 1 second,
which means a small network glitch (or system load?  maybe I'm all wrong
about this system never breaking a sweat, at least when it comes to
network traffic) would flip it, so I've increased it to 10 on both
machines (and apparently just induced a flip of my own. oops).  By the
nature of this system, some people will be annoyed by any flip, so it
really doesn't matter if it was a 1 second outage or a 30 second outage,
I just want the system available again after an unhappy event (or
routine maintenance).


the course adjustment in seconds is advbase, advskew is a much smaller
delay meant for a config with primary/backup where the backup advertises
just slightly less frequently.


Um. yeah.  I set advbase, and typed advskew in the e-mail. my bad.
After setting to 10, I have gone over two weeks without any flips, so that
looks like that is a pretty good fix.
 
Thanks for the guidance!


Nick.

Re: carp flapping

[Kapetanakis Giannis]

On 16/05/2023 00:11, Lyndon Nerenberg (VE7TFX/VE6BBM) wrote:

Nick, spare yourself the pain and just designate one machine as the
master.  This is how we run all our proxy server pairs (nginx,
squid, other stuff).  For a pair fooa/foob, 'a' is the master, and
gets advskew 100. The 'b' host gets 150. Make sure preemption is
enabled.

When it's upgrade time, upgrade the 'b' machine and reboot. If it
looks stable, set its advskew to 50 and wait for it to pick up
traffic.  Now upgrade and reboot the 'a' host. When it looks happy,
set 'b's advskew back to 150.

This keeps everything in a known state.  You are going to break
connections no matter what -- even when you let the master float
-- so you might as well do it under your own control.  We schedule
our updates for off-peak hours, and accept that the flip is going
to interrupt traffic.  You just have to live with it.

We moved to this scheme on all our proxies and firewalls seven
years ago and have never looked back.

--lyndon

Totally agree on this and on top of that add load balancers/routers in 
the mix which will run carp/relayd/pfsync/forwarding.


With sticky sessions, all requests will redirect to the same backend 
server and you can avoid breaking service connections.

These don't have to be big machines.

G

Re: carp flapping

[Lyndon Nerenberg (VE7TFX/VE6BBM)]

Nick, spare yourself the pain and just designate one machine as the
master.  This is how we run all our proxy server pairs (nginx,
squid, other stuff).  For a pair fooa/foob, 'a' is the master, and
gets advskew 100. The 'b' host gets 150. Make sure preemption is
enabled.

When it's upgrade time, upgrade the 'b' machine and reboot. If it
looks stable, set its advskew to 50 and wait for it to pick up
traffic.  Now upgrade and reboot the 'a' host. When it looks happy,
set 'b's advskew back to 150.

This keeps everything in a known state.  You are going to break
connections no matter what -- even when you let the master float
-- so you might as well do it under your own control.  We schedule
our updates for off-peak hours, and accept that the flip is going
to interrupt traffic.  You just have to live with it.

We moved to this scheme on all our proxies and firewalls seven
years ago and have never looked back.

--lyndon

Re: carp flapping

[Kapetanakis Giannis]

On 12/05/2023 14:43, Nick Holland wrote:
> I had several other people suggest network problems.  I'm not going to
> say "impossible" or even "unlikely", but my understanding is that the
> two machines are both plugged into the same switch, in the same rack.
>
> Several people pointed out I was using the default advskew of 1 second,
> which means a small network glitch (or system load?  maybe I'm all wrong
> about this system never breaking a sweat, at least when it comes to
> network traffic) would flip it, so I've increased it to 10 on both
> machines (and apparently just induced a flip of my own. oops).  By the
> nature of this system, some people will be annoyed by any flip, so it
> really doesn't matter if it was a 1 second outage or a 30 second outage,
> I just want the system available again after an unhappy event (or
> routine maintenance).
>
> Nick.

Usually it's a network problem. The big delay of 3 days you had also suggests 
that.

But on the other hand, I also had a similar problem in one of my load balancers 
(routing/fw/relayd), where the MASTER was becoming BACKUP for no obvious 
reason. I believed it was a network glitch, but couldn't trace it.

The problem after all was that they where pushing the limit of max pf states 
and relayd checks where failing. Not obvious to spot at all. I believe default 
is 20K.

pfctl -sm
pfctl -si

After increasing that limit with set limit states I've never had a glitch any 
more.

G

Re: carp flapping

[Stuart Henderson]

On 2023-05-12, Nick Holland  wrote:
> On 5/12/23 03:28, Stuart Henderson wrote:
>> On 2023-05-12, Nick Holland  wrote:
>>> Here's the problem I've seen:  I have my two machines flipping state
>>> randomly(?).  This bothers me because that means it is breaking  people's
>>> downloads.  Longest period betweek flips was less than two weeks.
>>>
>>> So ... I cranked up the carp logging to 5 and then 7 to see what it had
>>> to say about why...and it had almost nothing to say.
>> 
>> Does netstat -s -p carp give any enlightenment?
>
>
> ok, I just skewed the stats by taking the opportunity to bring the now
> backup up to -current, so node1 does not have the most recent flap:
>
> node1 $ uptime
>   7:18AM  up  8:22, 1 user, load averages: 0.00, 0.05, 0.08
>
> node1 $ doas netstat -s -p carp
> carp:
>  29981 packets received (IPv4)
>  0 packets received (IPv6)
>  0 packets discarded for bad interface
>  0 packets discarded for wrong TTL
>  0 packets shorter than header
>  0 discarded for bad checksums
>  0 discarded packets with a bad version
>  0 discarded because packet too short
>  0 discarded for bad authentication
>  0 discarded for unknown vhid
>  0 discarded because of a bad address list
>  0 packets sent (IPv4)
>  0 packets sent (IPv6)
>  0 send failed due to mbuf memory error
>  0 transitions to master
>
>   node2 $ uptime
>   7:19AM  up 4 days, 20:58, 2 users, load averages: 0.83, 0.78, 0.73
>
> $ ] netstat -s -p carp
> carp:
>  367836 packets received (IPv4)
>  0 packets received (IPv6)
>  0 packets discarded for bad interface
>  0 packets discarded for wrong TTL
>  0 packets shorter than header
>  0 discarded for bad checksums
>  0 discarded packets with a bad version
>  0 discarded because packet too short
>  0 discarded for bad authentication
>  0 discarded for unknown vhid
>  0 discarded because of a bad address list
>  52806 packets sent (IPv4)
>  0 packets sent (IPv6)
>  0 send failed due to mbuf memory error
>  2 transitions to master
>
>
> Will monitor going forward, though.
>
>
> I had several other people suggest network problems.  I'm not going to
> say "impossible" or even "unlikely", but my understanding is that the
> two machines are both plugged into the same switch, in the same rack.

You can also look at

netstat -ni -I ixl0
netstat -ni -I ixl0 -e
kstat ixl0:::

which may give some other clues

even pfctl -si might have something relevant

> Several people pointed out I was using the default advskew of 1 second,
> which means a small network glitch (or system load?  maybe I'm all wrong
> about this system never breaking a sweat, at least when it comes to
> network traffic) would flip it, so I've increased it to 10 on both
> machines (and apparently just induced a flip of my own. oops).  By the
> nature of this system, some people will be annoyed by any flip, so it
> really doesn't matter if it was a 1 second outage or a 30 second outage,
> I just want the system available again after an unhappy event (or
> routine maintenance).

the course adjustment in seconds is advbase, advskew is a much smaller
delay meant for a config with primary/backup where the backup advertises
just slightly less frequently.

Re: carp flapping

[Nick Holland]

On 5/12/23 03:28, Stuart Henderson wrote:

On 2023-05-12, Nick Holland  wrote:

Here's the problem I've seen:  I have my two machines flipping state
randomly(?).  This bothers me because that means it is breaking  people's
downloads.  Longest period betweek flips was less than two weeks.

So ... I cranked up the carp logging to 5 and then 7 to see what it had
to say about why...and it had almost nothing to say.


Does netstat -s -p carp give any enlightenment?



ok, I just skewed the stats by taking the opportunity to bring the now
backup up to -current, so node1 does not have the most recent flap:

node1 $ uptime
 7:18AM  up  8:22, 1 user, load averages: 0.00, 0.05, 0.08

node1 $ doas netstat -s -p carp
carp:
29981 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
0 discarded for unknown vhid
0 discarded because of a bad address list
0 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 transitions to master

 node2 $ uptime
 7:19AM  up 4 days, 20:58, 2 users, load averages: 0.83, 0.78, 0.73

$ ] netstat -s -p carp
carp:
367836 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
0 discarded for unknown vhid
0 discarded because of a bad address list
52806 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
2 transitions to master


Will monitor going forward, though.


I had several other people suggest network problems.  I'm not going to
say "impossible" or even "unlikely", but my understanding is that the
two machines are both plugged into the same switch, in the same rack.

Several people pointed out I was using the default advskew of 1 second,
which means a small network glitch (or system load?  maybe I'm all wrong
about this system never breaking a sweat, at least when it comes to
network traffic) would flip it, so I've increased it to 10 on both
machines (and apparently just induced a flip of my own. oops).  By the
nature of this system, some people will be annoyed by any flip, so it
really doesn't matter if it was a 1 second outage or a 30 second outage,
I just want the system available again after an unhappy event (or
routine maintenance).

Nick.

Re: carp flapping

[Stuart Henderson]

On 2023-05-12, Nick Holland  wrote:
> Here's the problem I've seen:  I have my two machines flipping state
> randomly(?).  This bothers me because that means it is breaking  people's
> downloads.  Longest period betweek flips was less than two weeks.
>
> So ... I cranked up the carp logging to 5 and then 7 to see what it had
> to say about why...and it had almost nothing to say.

Does netstat -s -p carp give any enlightenment?

Re: carp status master on both firewalls

[mabi]

--- Original Message ---
On Friday, April 14th, 2023 at 7:14 AM, Janne Johansson  
wrote:

> Not impossible to have switches(*) that dislike/filter/bug on
> multicast too I guess, so I would suggest rigging the carps up (at
> least temporary) with carppeer against the "real" ip of the remote
> ext_if to make carp use normal unicast ip for sync and just see if it
> helps. If it does, it is related to the boxes ability to talk
> multicast and you would have to either stick with carppeer setup, or
> "fix" the multicast issue, which can be hard to pin down where exactly
> it is.

Thank you Janne for pointing out the switch. I would have never thought about 
that. So indeed, I just upgraded my Cisco Catalyst 2960L switch the latest IOS 
version of 2022 and now the 2nd firewall correctly reports backup as status. I 
was running an IOS version from 2018. Strangely enough both firewalls are 
connected to that switch with both carp0 to the public VLAN and both carp1 to 
the private VLAN so I would have expected the same odd double master status on 
both carp interfaces and not just on carp0. But anyway it works now.

Re: carp status master on both firewalls

[mabi]

--- Original Message ---
On Friday, April 14th, 2023 at 10:50 AM, Markus Wernig  
wrote:

Thank you Markus for your answer, as mentioned to Janne it was the switch the 
problem. For the sake of documenting I answered your questions below.

> - Do the two fw actually have a link on their carp0 carpdev interfaces?

Yes.

> If both are master, both should be sending out CARP advertisements, so
> I'd try to run tcpdump on both external interfaces and look for those:
> tcpdump -n -e -i carp0 proto carp

I did that yesterday and for both firewalls I could see the CARPv2 
advertisements.

> - Did you enable CARP preemption? Try setting these via sysctl:
> net.inet.carp.preempt=1
> net.inet.carp.log=3

I have CARP preemption enabled but my carp log level is 2 and not 3.

> - In your config one fw has carpdev em2, the other carpdev em0. Could be
> OK, or could be an error.

Well spotted but indeed it is correct, both firewalls have different hardware 
and the first interface on the first firewall is em2 whereas on the 2nd 
firewall it is em0.

Re: carp status master on both firewalls

[Markus Wernig]

for my external carp interface both firewalls show master as status 


The config is below for reference:

/etc/hostname.carp0 on fw1

inet x.x.x.114 255.255.255.240 x.x.x.127 vhid 40 carpdev em2 pass password 
advskew 1
inet alias x.x.x.115 0xfff0
inet alias x.x.x.116 0xfff0

/etc/hostname.carp0 on fw2
  
inet x.x.x.114 255.255.255.240 x.x.x.127 vhid 40 carpdev em0 pass password advskew 128

inet alias x.x.x.115 0xfff0
inet alias x.x.x.116 0xfff0

On both firewalls I have added the following in /etc/pf.conf:

pass on { $ext_if $int_if } proto carp keep state (no-sync)

Did anyone already encounter this issue or has any idea what might be wrong?


Hard to tell without logs. Some things that come to mind:

- Do the two fw actually have a link on their carp0 carpdev interfaces? 
If both are master, both should be sending out CARP advertisements, so 
I'd try to run tcpdump on both external interfaces and look for those:

tcpdump -n -e -i carp0 proto carp

- Did you enable CARP preemption? Try setting these via sysctl:
net.inet.carp.preempt=1
net.inet.carp.log=3

- In your config one fw has carpdev em2, the other carpdev em0. Could be 
OK, or could be an error.

Re: CARP and DHCP

[Christer Solskogen]

On Sun, Jan 8, 2023 at 5:23 PM Nick Holland 
wrote:

>
> Does this actually maintain state?  I'm thinking pfsync might
> not work properly when the external interface "changes" like that.
> It wouldn't actually matter much in *my case*, but I'm wondering
> about the more general case.
>
>
>
I no expert, but I think so. I can be in a Teams meeting while I switch
from master to slave, without loosing the connection.

Re: CARP and DHCP

[Nick Holland]

On 1/6/23 02:31, Christer Solskogen wrote:

On Mon, Jan 2, 2023 at 5:14 PM Nick Holland 
wrote:


hiya.

Goal: home (i.e., DHCP external network config) redundant
firewalls with CARP and PFSYNC.





Totally doable. I've been running it like that for the last 7 years at
home.
My ISP doesn't like it when the two firewalls have different mac-addresses,


same here. :)


so I have to do some spoofing on the slave machine.
ifstated is your very good friend here.  My /etc/hostname.$extif is empty.

CARP is only in use for the internal interface.

This if my ifstated.conf on mster:

carp_up = "carp0.link.up"
carp_down = "!carp0.link.up"
carp_init = "carp0.link.unknown"

init-state auto

state auto {
 if ($carp_up)
 set-state fw_master
 if !($carp_up)
 set-state fw_slave
}

state fw_master {
 init {
 run "route -qn flush"
 run "ifconfig em2 inet autoconf"
 run "pfctl -f /etc/pf.conf"
 }

 if ($carp_down)
 set-state fw_slave
 if ($carp_init)
 run "sleep 2"
}

state fw_slave {
 init {
 run "ifconfig em2 -inet"
 run "route -qn flush"
 run "route add default 192.168.0.3"
 }

 if ($carp_up)
 set-state fw_master
}


Does this actually maintain state?  I'm thinking pfsync might
not work properly when the external interface "changes" like that.
It wouldn't actually matter much in *my case*, but I'm wondering
about the more general case.

Thanks!

Nick.

Re: CARP and DHCP

[Christer Solskogen]

On Mon, Jan 2, 2023 at 5:14 PM Nick Holland 
wrote:

> hiya.
>
> Goal: home (i.e., DHCP external network config) redundant
> firewalls with CARP and PFSYNC.
>
>


Totally doable. I've been running it like that for the last 7 years at
home.
My ISP doesn't like it when the two firewalls have different mac-addresses,
so I have to do some spoofing on the slave machine.
ifstated is your very good friend here.  My /etc/hostname.$extif is empty.

CARP is only in use for the internal interface.

This if my ifstated.conf on mster:

carp_up = "carp0.link.up"
carp_down = "!carp0.link.up"
carp_init = "carp0.link.unknown"

init-state auto

state auto {
if ($carp_up)
set-state fw_master
if !($carp_up)
set-state fw_slave
}

state fw_master {
init {
run "route -qn flush"
run "ifconfig em2 inet autoconf"
run "pfctl -f /etc/pf.conf"
}

if ($carp_down)
set-state fw_slave
if ($carp_init)
run "sleep 2"
}

state fw_slave {
init {
run "ifconfig em2 -inet"
run "route -qn flush"
run "route add default 192.168.0.3"
}

if ($carp_up)
set-state fw_master
}

Re: carp question

[Christer Solskogen]

On Tue, Jun 28, 2022 at 2:58 PM Stuart Henderson 
wrote:

>
> So for this you would need to monitor the interface status and change
> the default route, you couldn't rely on /etc/mygate.
>
>
I don't. I use ifstated :-)

Re: carp question

[Stuart Henderson]

On 2022-06-28, Christer Solskogen  wrote:
> On Tue, Jun 28, 2022 at 12:18 PM Łukasz Moskała  wrote:
>
>> You wanted to set the CARP IP as default gateway on both master and
>> backup, right?
>>
>>
> No, the master is the gateway. So what would not make sense as all.
> I would like to have the CARP IP as default gateway on the backup. (And
> vice-versa, when the now-master becomes backup)

So for this you would need to monitor the interface status and change
the default route, you couldn't rely on /etc/mygate.

But, if you do that anyway, you can just use the standard non-carp
address on the backup, and the proper upstream router on the active machine

Re: carp question

[Christer Solskogen]

On Tue, Jun 28, 2022 at 12:18 PM Łukasz Moskała  wrote:

> You wanted to set the CARP IP as default gateway on both master and
> backup, right?
>
>
No, the master is the gateway. So what would not make sense as all.
I would like to have the CARP IP as default gateway on the backup. (And
vice-versa, when the now-master becomes backup)

Re: carp question

[Łukasz Moskała]

Dnia Tue, Jun 28, 2022 at 11:36:55AM +0200, Christer Solskogen napisał(a):
> On Tue, Jun 28, 2022 at 10:44 AM Stuart Henderson 
> wrote:
> 
> >
> > It makes no sense to set your own address as the default gateway?
> >
> >
> It would *if* backup didn't respond to the carp IP.  But it does, so no it
> does not make sense.
> The reason I was hoping it would work was the wording in the FAQ.

You wanted to set the CARP IP as default gateway on both master and backup, 
right?

Then, even if it would work as you thought, what would happen on master?
You'd have CARP IP set as a gateway. The same CARP IP that is on this server.

Equivalent of setting 127.0.0.1 as a gateway.

Re: carp question

[Christer Solskogen]

On Tue, Jun 28, 2022 at 10:44 AM Stuart Henderson 
wrote:

>
> It makes no sense to set your own address as the default gateway?
>
>
It would *if* backup didn't respond to the carp IP.  But it does, so no it
does not make sense.
The reason I was hoping it would work was the wording in the FAQ.

Re: carp question

[Stuart Henderson]

On 2022-06-28, Christer Solskogen  wrote:
> On Tue, Jun 28, 2022 at 10:21 AM Łukasz Moskała  wrote:
>
>>
>> What problem are you trying to solve?
>>
>>
> Having identical config files on both the master and backup when it comes
> to setting up the default gateway. I was hoping I could just use the carp
> address as default gateway.

It makes no sense to set your own address as the default gateway?


-- 
Please keep replies on the mailing list.

Re: carp question

[Christer Solskogen]

On Tue, Jun 28, 2022 at 10:21 AM Łukasz Moskała  wrote:

>
> What problem are you trying to solve?
>
>
Having identical config files on both the master and backup when it comes
to setting up the default gateway. I was hoping I could just use the carp
address as default gateway.

-- 
chs

Re: carp question

[Łukasz Moskała]

Dnia Tue, Jun 28, 2022 at 10:03:25AM +0200, Christer Solskogen napisał(a):
> On Tue, Jun 28, 2022 at 9:52 AM Łukasz Moskała  wrote:
> 
> > Run tcpdump on master, ping on backup. If you see pings in tcpdump, then
> > master is responding.
> > If not, backup is responding to itself.
> >
> >
> Good catch.
> The backup is responding to it self. But should it? In the FAQ I find this:
> "The master host is the one that currently "holds" the shared IP; it
> responds to any traffic or ARP requests directed towards that address."
> So the question now is should master respond?
> 
> -- 
> chs

Master can't respond to that traffic, as it never goes over wire. Just like 
pinging 127.0.0.1

If you have 192.0.2.10/24 on em0, and ping 192.0.2.10 from host that has this 
IP, this traffic newer goes over network cable either.

Also this allows you to bind service to that CARP IP on both hosts. If that 
address would "disappear" from interface when host is in backup mode, services 
running on backup host couldn't bind to that IP.
For example you have CARP IP 192.0.2.200/32. You can bind httpd to 
192.0.2.200:443 on both hosts, and have both of them running, but only master 
would receive traffic.

Master is the only one who advertises this IP on the network. So from 
perspective of any other host, only master has this address.

Maybe you can get userspace CARP implementation that removes IP from interface 
when in backup mode and adds it again when in master mode. This is how 
keepalived on linux works (but it uses VRRP, not CARP).

What problem are you trying to solve?

--
Łukasz Moskała

Re: carp question

[Christer Solskogen]

On Tue, Jun 28, 2022 at 9:52 AM Łukasz Moskała  wrote:

> Run tcpdump on master, ping on backup. If you see pings in tcpdump, then
> master is responding.
> If not, backup is responding to itself.
>
>
Good catch.
The backup is responding to it self. But should it? In the FAQ I find this:
"The master host is the one that currently "holds" the shared IP; it
responds to any traffic or ARP requests directed towards that address."
So the question now is should master respond?

-- 
chs

Re: carp question

[Łukasz Moskała]

Dnia Tue, Jun 28, 2022 at 09:47:40AM +0200, Christer Solskogen napisał(a):
> if you ping the carp ip from the backup, does the master respond or the
> backup it self?
> 
> -- 
> chs

Hi,

Run tcpdump on master, ping on backup. If you see pings in tcpdump, then master 
is responding.
If not, backup is responding to itself.

--
Łukasz Moskała

Re: CARP Cold Spare

[Nick Holland]

On 9/24/21 6:13 PM, Don Tek wrote:

Would there be any ‘problem’ with configuring a 2-machine CARP
setup and then just keeping one machine powered-off until needed?

I realize this defeats live failover, but this is not a requirement
for my customer.

I just want them to be able to, in the event of a primary machine
failure, power-on the secondary and have it take over.  Logic here is
to otherwise not have the secondary sucking power off the UPS’s in
the event of a power failure, or in general.

Legit?



Technically, should work fine.
Administratively?  you got a mess on your hands.

With a fair amount of certainty, I can say you will run into at least
the following administrative problems:
* users added/changed to live box, not duplicated to spare.
* Updates done to live box, not to spare
* rule changes done to live box, not to spare.
* Other settings changed on live box, not duplicated to spare.

None of these are intrinsically CARP issues or solved by CARP at all
but all of them are are going to be complicated by having a machine that
is off when changes are made to the live one.  All these issues have to
be considered with a CARP setup, but with a machine powered off, you KNOW
they won't be dealt with in a timely manner...which means they won't be
there when you need them.

If your goal is really to have one machine running, I'd suggest skipping
CARP, and just mirror the drives on the primary and keep spare hw in
reserve, and keep good backups of ALL configuration information.  Drive
fails?  you got a mirror.  HW fails?  you got spare, move the drives,
bring it up.  Keep it simple, you will be happier.

Nick.

Re: CARP Cold Spare

[Stuart Henderson]

On 2021-09-25, leonard  wrote:
> What is the power draw? I use a 1500 VA apc backups with 6 outlets on ups and 
> 5 on surge protection. As long as your total draw is less than 1200 VA, for < 
> $200 canadian you have a cheap simple solution. Just put on on the ups side 
> and the other on the surge suppressor side. Or buy 2.leonard@on the road

Then, when the UPS is depleted and auto shutdown either failed or
wasn't setup in the first place, you can have two machines failing to
start due to fsck failures, not just one!

> > They experience multi-hour blackouts what seems like once a month;

If they aren't spending the money on lower power servers, the chance
of spending the money on a UPS and battery chain (or more likely,
ATS and generator) capable of surviving multi-hour blackouts is slim.

Seems to me the "cold spare" idea makes complete sense for the
situation described. I would not bother with pfsync for this use.
However, if the network config allows, I _would_ try to get some
low power box (rpi or whatever) connected to serial console
on both routers.

-- 
Please keep replies on the mailing list.

Re: CARP Cold Spare

[leonard]

What is the power draw? I use a 1500 VA apc backups with 6 outlets on ups and 5 
on surge protection. As long as your total draw is less than 1200 VA, for < 
$200 canadian you have a cheap simple solution. Just put on on the ups side and 
the other on the surge suppressor side. Or buy 2.leonard@on the road
 Original message From: Don Tek  Date: 
2021-09-25  11:40  (GMT-05:00) To: jslee  Cc: 
misc@openbsd.org Subject: Re: CARP Cold Spare I'm not sure why the hardware 
matters, but the two machines are a couple HP 1U Gen 8 Xeon servers.  Suffice 
to say, they are identical and have supported hardware configurations for 
OpenBSD.Of course I _could_ run one off direct power, but it would be a 
terrible idea.  The location is notorious for power surges, blips that are 
enough to reboot servers and several-second brown-outs.  So, not connected to 
the UPS is just asking for damages.They experience multi-hour blackouts what 
seems like once a month; this is where the desire to limit the draw on the 
UPS's comes from.  To ensure we make it through without having to shut 
down.Remote access is of primary concern, both for me for support, since I'm 
geographically far enough away that being on-site is not feasible, and to the 
customer, who just wants to stay home and work on systems in the 
office.Configurations on the servers almost never change (simple firwall), so 
besides having to run a quick syspatch and reboot once at time of failover, I 
don't see maintenance being so bad.  I keep config files backed-up otherwise 
centrally for quick restore to the running box as well.My primary concern here 
is if CARP / pfsync will have issues with the one machine being down a majority 
of the time.  Based on the FAQ, I think not, but have no practical experience.> 
On Sep 25, 2021, at 3:00 AM, jslee  wrote:> > Hi,> > 
You haven’t said anything about your hardware platform, but could you run one 
of them on non-UPS power? Then you’d still have one online when (*not* if) the 
UPS fails, and also they’ll both normally be online for maintenance, syspatch, 
config changes etc> > I do recall installing a pair of identical servers at the 
same time and having them both fail a year later within an hour of each other, 
both with seized CPU fans, so I am somewhat sympathetic to your idea. But I 
think the practical cost of maintenance may be rather high> > John> > >> On 
Sat, 25 Sep 2021, at 08:13, Don Tek wrote:>> Would there be any ‘problem’ with 
configuring a 2-machine CARP setup >> and then just keeping one machine 
powered-off until needed?>> >> I realize this defeats live failover, but this 
is not a requirement for >> my customer.>> >> I just want them to be able to, 
in the event of a primary machine >> failure, power-on the secondary and have 
it take over.  Logic here is >> to otherwise not have the secondary sucking 
power off the UPS’s in the >> event of a power failure, or in general.>> >> 
Legit?

Re: CARP Cold Spare

[Don Tek]

Unsure what the power draw is on these guys yet, they just got them.  They have 
redundant 450W Platinum power supplies.

The "new" servers are completely overkill for the application, but this is a 
work-with-what's-available situation.  They got these free from a friend and 
don't want to spend on new hardware, otherwise I'd just get them something 
"smaller" and supremely efficient.

We already have redundant UPS's.  With the 3 servers previously (main VM 
servers and 1 now-dead firewall) attached to battery-backed power, we were 
getting roughly 3.5 hours of runtime before UPS drop.  Sometimes minutes can 
matter with these blackouts.

I also want that with the one machine not running, it's not incurring any wear 
or tear just for the sake of hot failover.

I've got no talent on site, so in the event of a hardware or file system 
failure, It's an extremely tedious experience for me to walk someone there 
through swapping hardware, wiring, or reinstalling an operating system.  So, 
I'm looking for that turn-this-one-off-and-this-one-on recovery option, without 
the need to fiddle with any hardware.  If this works the way I want, the only 
single point of failure will be their cable modem, and Comcast can handle 
replacing that.

Option B is to just have identical configurations and have them need to swap 
the network wiring as part of the failover.  Will still prevent the days of 
downtime we're incurring now due to this failure, but not as simple for them, 
if the CARP solution is solid.

> On Sep 25, 2021, at 10:25 AM, leonard  wrote:
> 
> 
> What is the power draw? I use a 1500 VA apc backups with 6 outlets on ups and 
> 5 on surge protection. As long as your total draw is less than 1200 VA, for < 
> $200 canadian you have a cheap simple solution. Just put on on the ups side 
> and the other on the surge suppressor side. Or buy 2.
> 
> 
> 
> leonard@on the road
> 
> 
>  Original message 
> From: Don Tek 
> Date: 2021-09-25 11:40 (GMT-05:00)
> To: jslee 
> Cc: misc@openbsd.org
> Subject: Re: CARP Cold Spare
> 
> I'm not sure why the hardware matters, but the two machines are a couple HP 
> 1U Gen 8 Xeon servers.  Suffice to say, they are identical and have supported 
> hardware configurations for OpenBSD.
> 
> Of course I _could_ run one off direct power, but it would be a terrible 
> idea.  The location is notorious for power surges, blips that are enough to 
> reboot servers and several-second brown-outs.  So, not connected to the UPS 
> is just asking for damages.
> 
> They experience multi-hour blackouts what seems like once a month; this is 
> where the desire to limit the draw on the UPS's comes from.  To ensure we 
> make it through without having to shut down.
> 
> Remote access is of primary concern, both for me for support, since I'm 
> geographically far enough away that being on-site is not feasible, and to the 
> customer, who just wants to stay home and work on systems in the office.
> 
> Configurations on the servers almost never change (simple firwall), so 
> besides having to run a quick syspatch and reboot once at time of failover, I 
> don't see maintenance being so bad.  I keep config files backed-up otherwise 
> centrally for quick restore to the running box as well.
> 
> My primary concern here is if CARP / pfsync will have issues with the one 
> machine being down a majority of the time.  Based on the FAQ, I think not, 
> but have no practical experience.
> 
> > On Sep 25, 2021, at 3:00 AM, jslee  wrote:
> > 
> > Hi,
> > 
> > You haven’t said anything about your hardware platform, but could you run 
> > one of them on non-UPS power? Then you’d still have one online when (*not* 
> > if) the UPS fails, and also they’ll both normally be online for 
> > maintenance, syspatch, config changes etc
> > 
> > I do recall installing a pair of identical servers at the same time and 
> > having them both fail a year later within an hour of each other, both with 
> > seized CPU fans, so I am somewhat sympathetic to your idea. But I think the 
> > practical cost of maintenance may be rather high
> > 
> > John
> > 
> > 
> >> On Sat, 25 Sep 2021, at 08:13, Don Tek wrote:
> >> Would there be any ‘problem’ with configuring a 2-machine CARP setup 
> >> and then just keeping one machine powered-off until needed?
> >> 
> >> I realize this defeats live failover, but this is not a requirement for 
> >> my customer.
> >> 
> >> I just want them to be able to, in the event of a primary machine 
> >> failure, power-on the secondary and have it take over.  Logic here is 
> >> to otherwise not have the secondary sucking power off the UPS’s in the 
> >> event of a power failure, or in general.
> >> 
> >> Legit?
> 

Re: CARP Cold Spare

[Don Tek]

I'm not sure why the hardware matters, but the two machines are a couple HP 1U 
Gen 8 Xeon servers.  Suffice to say, they are identical and have supported 
hardware configurations for OpenBSD.

Of course I _could_ run one off direct power, but it would be a terrible idea.  
The location is notorious for power surges, blips that are enough to reboot 
servers and several-second brown-outs.  So, not connected to the UPS is just 
asking for damages.

They experience multi-hour blackouts what seems like once a month; this is 
where the desire to limit the draw on the UPS's comes from.  To ensure we make 
it through without having to shut down.

Remote access is of primary concern, both for me for support, since I'm 
geographically far enough away that being on-site is not feasible, and to the 
customer, who just wants to stay home and work on systems in the office.

Configurations on the servers almost never change (simple firwall), so besides 
having to run a quick syspatch and reboot once at time of failover, I don't see 
maintenance being so bad.  I keep config files backed-up otherwise centrally 
for quick restore to the running box as well.

My primary concern here is if CARP / pfsync will have issues with the one 
machine being down a majority of the time.  Based on the FAQ, I think not, but 
have no practical experience.

> On Sep 25, 2021, at 3:00 AM, jslee  wrote:
> 
> Hi,
> 
> You haven’t said anything about your hardware platform, but could you run one 
> of them on non-UPS power? Then you’d still have one online when (*not* if) 
> the UPS fails, and also they’ll both normally be online for maintenance, 
> syspatch, config changes etc
> 
> I do recall installing a pair of identical servers at the same time and 
> having them both fail a year later within an hour of each other, both with 
> seized CPU fans, so I am somewhat sympathetic to your idea. But I think the 
> practical cost of maintenance may be rather high
> 
> John
> 
> 
>> On Sat, 25 Sep 2021, at 08:13, Don Tek wrote:
>> Would there be any ‘problem’ with configuring a 2-machine CARP setup 
>> and then just keeping one machine powered-off until needed?
>> 
>> I realize this defeats live failover, but this is not a requirement for 
>> my customer.
>> 
>> I just want them to be able to, in the event of a primary machine 
>> failure, power-on the secondary and have it take over.  Logic here is 
>> to otherwise not have the secondary sucking power off the UPS’s in the 
>> event of a power failure, or in general.
>> 
>> Legit?

Re: CARP Cold Spare

[jslee]

Hi,

You haven’t said anything about your hardware platform, but could you run one 
of them on non-UPS power? Then you’d still have one online when (*not* if) the 
UPS fails, and also they’ll both normally be online for maintenance, syspatch, 
config changes etc

I do recall installing a pair of identical servers at the same time and having 
them both fail a year later within an hour of each other, both with seized CPU 
fans, so I am somewhat sympathetic to your idea. But I think the practical cost 
of maintenance may be rather high

John


On Sat, 25 Sep 2021, at 08:13, Don Tek wrote:
> Would there be any ‘problem’ with configuring a 2-machine CARP setup 
> and then just keeping one machine powered-off until needed?
>
> I realize this defeats live failover, but this is not a requirement for 
> my customer.
>
> I just want them to be able to, in the event of a primary machine 
> failure, power-on the secondary and have it take over.  Logic here is 
> to otherwise not have the secondary sucking power off the UPS’s in the 
> event of a power failure, or in general.
>
> Legit?

Re: CARP Cold Spare

[Brian Brombacher]

> On Sep 24, 2021, at 6:16 PM, Don Tek  wrote:
> 
> Would there be any ‘problem’ with configuring a 2-machine CARP setup and 
> then just keeping one machine powered-off until needed?
> 
> I realize this defeats live failover, but this is not a requirement for my 
> customer.
> 
> I just want them to be able to, in the event of a primary machine failure, 
> power-on the secondary and have it take over.  Logic here is to otherwise not 
> have the secondary sucking power off the UPS’s in the event of a power 
> failure, or in general.
> 
> Legit?
> 

Sounds legit to me.  Let’s you share the IP safely and easily, up or down.

Re: carp backup and disconnecting ssh session

[Sebastian Benoit]

MJ J(mikedotjack...@gmail.com) on 2021.05.23 17:58:47 +0300:
> Hi,
> 
> I have a carp master and backup on a pair of one-armed Rapsberry Pi 4B
> devices (router1 and router2) and when I ssh to the backup using the
> carp IP as my gateway, it repeatedly throws me out after a few seconds
> with the message:
> 
> My laptop's network config:
> ---
> IP: 192.168.4.109
> Subnet mask: 255.255.255.0
> Gateway: 192.168.4.1
> 
> Both RPI4s are connected to switchports with packets tagged for VLANs
> 2,3,4,6 and the network devices don't have IP configuration -
> everything is configured on VLAN interfaces with the single parent
> interface bse0. CARP failover actually works as expected, but as
> mentioned I am unable to maintain an ssh session with the backup
> "router2" while using the carp IPs as my network gateway.
> 
> Network switch is a Zyxel GS1200-8 with firmware V2.00(ABME.0)C0. Loop
> prevention is enabled and I have also tested with it disabled to no
> avail.
> 
> What happens:
> ---
> $ ssh 10.0.1.101
> Last login: Sun May 23 17:44:21 2021 from 10.0.1.100
> OpenBSD 6.9 (GENERIC.MP) #1134: Sun Apr 18 01:53:35 MDT 2021
> router2#
> router2# client_loop: send disconnect: Broken pipe

you ssh from 192.168.4.109 to 10.0.1.101?

My best guess is that you have asymetric routing and your carp master
router1 only sees one direction of the traffic: 

laptop -> router1 -> router2
and
router2 -> laptop

because router2 has your laptop network locally on vlan6.

Solution: ssh to 192.168.4.3.

> 
> 
> Router 1 network config:
> ---
> router1# cat hostname.bse0
> up
> 
> router1# cat hostname.vlan2
> 172.16.1.6/24 172.16.1.255 parent bse0 vnetid 2 group PFSYNC
> description "private segment with router2"
> 
> router1# cat hostname.vlan3
> 10.0.1.100/24 10.0.1.255 parent bse0 vnetid 3 group INTERNAL
> description "router1 internal interface"
> 
> router1# cat hostname.vlan4
> 192.168.1.252/24 192.168.1.255 parent bse0 vnetid 4 group OLDSHIT
> description "unmigrated shit"
> 
> router1# cat hostname.vlan6
> 192.168.4.2/24 192.168.4.255 parent bse0 vnetid 6 group TCWIFI
> description "Time-Capsule Wifi"
> 
> router1# cat hostname.carp4
> 192.168.1.1/24 carpdev vlan4 pass fukdissh1t vhid 41 advskew 1
> description "TC-WIFI gateway"
> 
> router1# cat hostname.carp6
> 192.168.4.1/24 carpdev vlan6 pass fukdissh1t vhid 61 advskew 1
> description "TC-WIFI gateway"
> 
> 
> Router2 network config:
> ---
> router2# cat hostname.bse0
> up
> 
> router2# cat hostname.vlan2
> 172.16.1.7/24 172.16.1.255 parent bse0 vnetid 2 group PFSYNC
> description "private segment with router1"
> 
> router2# cat hostname.vlan3
> 10.0.1.101/24 10.0.1.255 parent bse0 vnetid 3 group INTERNAL
> description "router2 internal interface"
> 
> router2# cat hostname.vlan4
> 192.168.1.253/24 192.168.1.255 parent bse0 vnetid 4 group OLDSHIT
> description "unmigrated shit"
> 
> router2# cat hostname.vlan6
> 192.168.4.3/24 192.168.4.255 parent bse0 vnetid 6 group TCWIFI
> description "Time-Capsule Wifi"
> 
> router2# cat hostname.carp4
> 192.168.1.1/24 carpdev vlan4 pass fukdissh1t vhid 41 advskew 128
> description "TC-WIFI gateway"
> 
> router2# cat hostname.carp6
> 192.168.4.1/24 carpdev vlan6 pass fukdissh1t vhid 61 advskew 128
> description "TC-WIFI gateway"
> 
> 
> Any tips much appreciated.
> 
> -mike
> 

-- 

Re: CARP load balancing problems under KVM

[Carlos Lopez]

Many thanks for your help Giannis ... I am not using oVirt to manage this KVM 
host, only default installed tools: libvirtd, virsh ... In any case there is 
not any filter applied in libvirtd 

On 12/1/21, 20:13, "owner-m...@openbsd.org on behalf of Kapetanakis Giannis" 
 wrote:

On 12/01/2021 18:58, Carlos Lopez wrote:
> Thanks Gianni, but about what interface ? KVM bridges? In theory, MAC 
spoofing is avoided using this option:
>
> bridge.ageing-time: 300
>
> On 12/1/21, 17:47, "owner-m...@openbsd.org on behalf of Kapetanakis 
Giannis"  wrote:
>
>  Check that you have mac spoofing filter disabled on that interface.


For carp to work, I have it disabled on the virtual interface on the 
supervisor.

I use ovirt so I do it on the network profile there.

This is a libvirt filter option. Maybe it's applied by default on your 
setup. The idea is to disable this protection and allow mac spoofing.

G

Re: CARP load balancing problems under KVM

[Kapetanakis Giannis]

On 12/01/2021 18:58, Carlos Lopez wrote:

Thanks Gianni, but about what interface ? KVM bridges? In theory, MAC spoofing 
is avoided using this option:

bridge.ageing-time: 300

On 12/1/21, 17:47, "owner-m...@openbsd.org on behalf of Kapetanakis Giannis" 
 wrote:

 Check that you have mac spoofing filter disabled on that interface.



For carp to work, I have it disabled on the virtual interface on the 
supervisor.


I use ovirt so I do it on the network profile there.

This is a libvirt filter option. Maybe it's applied by default on your 
setup. The idea is to disable this protection and allow mac spoofing.


G

Re: CARP load balancing problems under KVM

[Carlos Lopez]

Thanks Gianni, but about what interface ? KVM bridges? In theory, MAC spoofing 
is avoided using this option:

bridge.ageing-time: 300

On 12/1/21, 17:47, "owner-m...@openbsd.org on behalf of Kapetanakis Giannis" 
 wrote:

Check that you have mac spoofing filter disabled on that interface.

G

On 12/01/2021 15:30, Carlos Lopez wrote:
> Hi David and misc@,
>
> Sorry to disturb with this.I have realized several tests this morning 
with two OpenBSD 6.8 carp'ed firewalls (fully patched) as kvm guests and result 
is the same: carp load balancing doesn't work. My host is a RedHat Enterprise 
Linux 8.3 with kernel .18.0-240.10.1.el8_3.x86_6 (fully patched also). I have 
tested all ip load balancing options under these OpenBSD virtual guests: ip, 
ip-unicast and ip-stealth.
>
> When I use only "ip" for load balancing in all carped interfaces, these 
appears as MASTER in both firewalls as you can see here:
>
> carp0: flags=8843 mtu 1500
>   lladdr 00:00:5e:00:01:0a
>   description: Production Network
>   index 12 priority 15 llprio 3
>   carp: carpdev vio0 advbase 1 balancing ip
>   state MASTER vhid 10 advskew 0
>   state MASTER vhid 11 advskew 100
>   groups: carp
>   status: master
>   inet 172.22.55.30 netmask 0xffe0 broadcast 172.22.55.31
> carp1: flags=8843 mtu 1500
>   lladdr 00:00:5e:00:01:0c
>   description: Primary Public Network
>   index 13 priority 15 llprio 3
>   carp: carpdev vio1 advbase 1 balancing ip
>   state MASTER vhid 12 advskew 0
>   state MASTER vhid 13 advskew 100
>   groups: carp
>   status: master
>   inet 172.17.35.5 netmask 0xff00 broadcast 172.17.35.255
> carp2: flags=8843 mtu 1500
>   lladdr 00:00:5e:00:01:0e
>   description: Mgmt Network
>   index 14 priority 15 llprio 3
>   carp: carpdev vio2 advbase 1 balancing ip
>   state MASTER vhid 14 advskew 0
>   state MASTER vhid 15 advskew 100
>   groups: carp
>   status: master
>   inet 172.22.59.1 netmask 0xfff0 broadcast 172.22.59.15
> carp3: flags=8843 mtu 1500
>   lladdr 00:00:5e:00:01:10
>   description: DMZ Network
>   index 15 priority 15 llprio 3
>   carp: carpdev vio3 advbase 1 balancing ip
>   state MASTER vhid 16 advskew 0
>   state MASTER vhid 17 advskew 100
>   groups: carp
>   status: master
>   inet 172.22.54.1 netmask 0xfff8 broadcast 172.22.54.7
> carp4: flags=8843 mtu 1500
>   lladdr 00:00:5e:00:01:12
>   description: VPN Network
>   index 16 priority 15 llprio 3
>   carp: carpdev vio4 advbase 1 balancing ip
>   state MASTER vhid 18 advskew 0
>   state MASTER vhid 19 advskew 100
>   groups: carp
>   status: master
>   inet 172.22.56.1 netmask 0xfff8 broadcast 172.22.56.7
> carp5: flags=8843 mtu 1500
>   lladdr 00:00:5e:00:01:14
>   description: Encryption Network
>   index 17 priority 15 llprio 3
>   carp: carpdev vio5 advbase 1 balancing ip
>   state MASTER vhid 20 advskew 0
>   state MASTER vhid 21 advskew 100
>   groups: carp
>   status: master
>   inet 172.22.57.1 netmask 0xfff8 broadcast 172.22.57.7
> carp6: flags=8843 mtu 1500
>   lladdr 00:00:5e:00:01:16
>   description: IDPS Network
>   index 18 priority 15 llprio 3
>   carp: carpdev vio6 advbase 1 balancing ip
>   state MASTER vhid 22 advskew 0
>   state MASTER vhid 23 advskew 100
>   groups: carp
>   status: master
>   inet 172.22.60.1 netmask 0xfff0 broadcast 172.22.60.15
> carp7: flags=8843 mtu 1500
>   lladdr 00:00:5e:00:01:18
>   description: Windows Network
>   index 19 priority 15 llprio 3
>   carp: carpdev vio8 advbase 1 balancing ip
>   state MASTER vhid 24 advskew 0
>   state MASTER vhid 25 advskew 100
>   groups: carp
>   status: master
>   inet 172.22.61.1 netmask 0xfff8 broadcast 172.22.61.7
>
> .. and all kvm guests behind these firewalls shows the following arp 
entries (172.22.55.30 is the carp'ed IP address. IP 172.22.55.28 is for fw01 
and 172.22.55.29 is for fw02):
>
> root@stonehenge:~# ip neigh
> 172.22.55.14 dev eth0 lladdr ac:1f:6b:14:6a:f6 STALE
> 172.22.55.29 dev eth0 lladdr 00:50:56:a1:4d:c3 STALE
> 172.22.55.5 dev eth0 lladdr ac:1f:6b:14:6c:b0 STALE
> 172.22.55.30 dev eth0  INCOMPLETE
> 172.22.55.28 dev eth0 lladdr 00:50:56:6f:64:aa STALE
> 172.22.55.4 dev eth0 lladdr a0:ce:c8:0a:7e:f4 REACHABLE
>
> And after some seconds, arp entries appears as:
>
> root@stonehenge:~# ip neigh
> 172.22.55.14 dev eth0 lladdr ac:1f:6b:14:6a:f6 STALE
> 172.22.55.29 dev eth0 lladdr 00:50:56:a1:4d:c3 STALE
> 172.22.55.5 dev eth0 lladdr ac:1f:6b:14:6c:b0 STALE

Re: CARP load balancing problems under KVM

[Kapetanakis Giannis]

Check that you have mac spoofing filter disabled on that interface.

G

On 12/01/2021 15:30, Carlos Lopez wrote:

Hi David and misc@,

Sorry to disturb with this.I have realized several tests this morning with two 
OpenBSD 6.8 carp'ed firewalls (fully patched) as kvm guests and result is the 
same: carp load balancing doesn't work. My host is a RedHat Enterprise Linux 
8.3 with kernel .18.0-240.10.1.el8_3.x86_6 (fully patched also). I have tested 
all ip load balancing options under these OpenBSD virtual guests: ip, 
ip-unicast and ip-stealth.

When I use only "ip" for load balancing in all carped interfaces, these appears 
as MASTER in both firewalls as you can see here:

carp0: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0a
description: Production Network
index 12 priority 15 llprio 3
carp: carpdev vio0 advbase 1 balancing ip
state MASTER vhid 10 advskew 0
state MASTER vhid 11 advskew 100
groups: carp
status: master
inet 172.22.55.30 netmask 0xffe0 broadcast 172.22.55.31
carp1: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0c
description: Primary Public Network
index 13 priority 15 llprio 3
carp: carpdev vio1 advbase 1 balancing ip
state MASTER vhid 12 advskew 0
state MASTER vhid 13 advskew 100
groups: carp
status: master
inet 172.17.35.5 netmask 0xff00 broadcast 172.17.35.255
carp2: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0e
description: Mgmt Network
index 14 priority 15 llprio 3
carp: carpdev vio2 advbase 1 balancing ip
state MASTER vhid 14 advskew 0
state MASTER vhid 15 advskew 100
groups: carp
status: master
inet 172.22.59.1 netmask 0xfff0 broadcast 172.22.59.15
carp3: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:10
description: DMZ Network
index 15 priority 15 llprio 3
carp: carpdev vio3 advbase 1 balancing ip
state MASTER vhid 16 advskew 0
state MASTER vhid 17 advskew 100
groups: carp
status: master
inet 172.22.54.1 netmask 0xfff8 broadcast 172.22.54.7
carp4: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:12
description: VPN Network
index 16 priority 15 llprio 3
carp: carpdev vio4 advbase 1 balancing ip
state MASTER vhid 18 advskew 0
state MASTER vhid 19 advskew 100
groups: carp
status: master
inet 172.22.56.1 netmask 0xfff8 broadcast 172.22.56.7
carp5: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:14
description: Encryption Network
index 17 priority 15 llprio 3
carp: carpdev vio5 advbase 1 balancing ip
state MASTER vhid 20 advskew 0
state MASTER vhid 21 advskew 100
groups: carp
status: master
inet 172.22.57.1 netmask 0xfff8 broadcast 172.22.57.7
carp6: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:16
description: IDPS Network
index 18 priority 15 llprio 3
carp: carpdev vio6 advbase 1 balancing ip
state MASTER vhid 22 advskew 0
state MASTER vhid 23 advskew 100
groups: carp
status: master
inet 172.22.60.1 netmask 0xfff0 broadcast 172.22.60.15
carp7: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:18
description: Windows Network
index 19 priority 15 llprio 3
carp: carpdev vio8 advbase 1 balancing ip
state MASTER vhid 24 advskew 0
state MASTER vhid 25 advskew 100
groups: carp
status: master
inet 172.22.61.1 netmask 0xfff8 broadcast 172.22.61.7

.. and all kvm guests behind these firewalls shows the following arp entries 
(172.22.55.30 is the carp'ed IP address. IP 172.22.55.28 is for fw01 and 
172.22.55.29 is for fw02):

root@stonehenge:~# ip neigh
172.22.55.14 dev eth0 lladdr ac:1f:6b:14:6a:f6 STALE
172.22.55.29 dev eth0 lladdr 00:50:56:a1:4d:c3 STALE
172.22.55.5 dev eth0 lladdr ac:1f:6b:14:6c:b0 STALE
172.22.55.30 dev eth0  INCOMPLETE
172.22.55.28 dev eth0 lladdr 00:50:56:6f:64:aa STALE
172.22.55.4 dev eth0 lladdr a0:ce:c8:0a:7e:f4 REACHABLE

And after some seconds, arp entries appears as:

root@stonehenge:~# ip neigh
172.22.55.14 dev eth0 lladdr ac:1f:6b:14:6a:f6 STALE
172.22.55.29 dev eth0 lladdr 00:50:56:a1:4d:c3 STALE
172.22.55.5 dev eth0 lladdr ac:1f:6b:14:6c:b0 STALE
172.22.55.30 dev eth0  FAILED
172.22.55.28 dev eth0 lladdr 00:50:56:6f:64:aa REACHABLE
172.22.55.4 dev eth0 lladdr a0:ce:c8:0a:7e:f4 DELAY

In this case, no traffic flows and I can't see any packet using tcpdump 

When I use "ip-unicast" or "ip-stealth" options for CARP load balancing, CARP 
interfaces seems to be ok:

carp0: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0a
description: Production Network

Re: CARP load balancing problems under KVM

[Carlos Lopez]

Hi David and misc@,

Sorry to disturb with this.I have realized several tests this morning with two 
OpenBSD 6.8 carp'ed firewalls (fully patched) as kvm guests and result is the 
same: carp load balancing doesn't work. My host is a RedHat Enterprise Linux 
8.3 with kernel .18.0-240.10.1.el8_3.x86_6 (fully patched also). I have tested 
all ip load balancing options under these OpenBSD virtual guests: ip, 
ip-unicast and ip-stealth.

When I use only "ip" for load balancing in all carped interfaces, these appears 
as MASTER in both firewalls as you can see here:

carp0: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0a
description: Production Network
index 12 priority 15 llprio 3
carp: carpdev vio0 advbase 1 balancing ip
state MASTER vhid 10 advskew 0
state MASTER vhid 11 advskew 100
groups: carp
status: master
inet 172.22.55.30 netmask 0xffe0 broadcast 172.22.55.31
carp1: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0c
description: Primary Public Network
index 13 priority 15 llprio 3
carp: carpdev vio1 advbase 1 balancing ip
state MASTER vhid 12 advskew 0
state MASTER vhid 13 advskew 100
groups: carp
status: master
inet 172.17.35.5 netmask 0xff00 broadcast 172.17.35.255
carp2: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0e
description: Mgmt Network
index 14 priority 15 llprio 3
carp: carpdev vio2 advbase 1 balancing ip
state MASTER vhid 14 advskew 0
state MASTER vhid 15 advskew 100
groups: carp
status: master
inet 172.22.59.1 netmask 0xfff0 broadcast 172.22.59.15
carp3: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:10
description: DMZ Network
index 15 priority 15 llprio 3
carp: carpdev vio3 advbase 1 balancing ip
state MASTER vhid 16 advskew 0
state MASTER vhid 17 advskew 100
groups: carp
status: master
inet 172.22.54.1 netmask 0xfff8 broadcast 172.22.54.7
carp4: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:12
description: VPN Network
index 16 priority 15 llprio 3
carp: carpdev vio4 advbase 1 balancing ip
state MASTER vhid 18 advskew 0
state MASTER vhid 19 advskew 100
groups: carp
status: master
inet 172.22.56.1 netmask 0xfff8 broadcast 172.22.56.7
carp5: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:14
description: Encryption Network
index 17 priority 15 llprio 3
carp: carpdev vio5 advbase 1 balancing ip
state MASTER vhid 20 advskew 0
state MASTER vhid 21 advskew 100
groups: carp
status: master
inet 172.22.57.1 netmask 0xfff8 broadcast 172.22.57.7
carp6: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:16
description: IDPS Network
index 18 priority 15 llprio 3
carp: carpdev vio6 advbase 1 balancing ip
state MASTER vhid 22 advskew 0
state MASTER vhid 23 advskew 100
groups: carp
status: master
inet 172.22.60.1 netmask 0xfff0 broadcast 172.22.60.15
carp7: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:18
description: Windows Network
index 19 priority 15 llprio 3
carp: carpdev vio8 advbase 1 balancing ip
state MASTER vhid 24 advskew 0
state MASTER vhid 25 advskew 100
groups: carp
status: master
inet 172.22.61.1 netmask 0xfff8 broadcast 172.22.61.7

.. and all kvm guests behind these firewalls shows the following arp entries 
(172.22.55.30 is the carp'ed IP address. IP 172.22.55.28 is for fw01 and 
172.22.55.29 is for fw02):

root@stonehenge:~# ip neigh
172.22.55.14 dev eth0 lladdr ac:1f:6b:14:6a:f6 STALE
172.22.55.29 dev eth0 lladdr 00:50:56:a1:4d:c3 STALE
172.22.55.5 dev eth0 lladdr ac:1f:6b:14:6c:b0 STALE
172.22.55.30 dev eth0  INCOMPLETE
172.22.55.28 dev eth0 lladdr 00:50:56:6f:64:aa STALE
172.22.55.4 dev eth0 lladdr a0:ce:c8:0a:7e:f4 REACHABLE

And after some seconds, arp entries appears as:

root@stonehenge:~# ip neigh
172.22.55.14 dev eth0 lladdr ac:1f:6b:14:6a:f6 STALE
172.22.55.29 dev eth0 lladdr 00:50:56:a1:4d:c3 STALE
172.22.55.5 dev eth0 lladdr ac:1f:6b:14:6c:b0 STALE
172.22.55.30 dev eth0  FAILED
172.22.55.28 dev eth0 lladdr 00:50:56:6f:64:aa REACHABLE
172.22.55.4 dev eth0 lladdr a0:ce:c8:0a:7e:f4 DELAY

In this case, no traffic flows and I can't see any packet using tcpdump 

When I use "ip-unicast" or "ip-stealth" options for CARP load balancing, CARP 
interfaces seems to be ok:

carp0: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0a
description: Production Network
index 12 priority 15 llprio 3
carp: carpdev vio0 advbase 1 balancing ip-stealth
state MASTER 

Re: CARP load balancing problems under KVM

[Uwe Werler]

On 21 Oct 07:12, Carlos Lopez wrote:
> Hi all,
> 
> Before upgrade from OpenBSD 6.7 to OpenBSD 6.8, my pair firewalls was using 
> carp in IP balance mode without problems from several months. These firewalls 
> are installed in a RHEL 8.2 (fully patched) KVM host.
> 
> After upgrading to OpenBSD 6.8, carp ip balance mode doesn’t works. I have 
> tested reconfiguring balance mode for ip-stealth and ip-unicast also and the 
> result is always the same: network packets are not processed by firewalls. 
> But if I configure CARP using “the simple configuration” and one node is 
> master and the other is backup all it is working without problems.
> 
> Since the configuration is exactly the same for version 6.8 as it was for 
> version 6.7, could there be some kind of problem with the virtio driver on 
> the network interfaces in version 6.8?
> 
> Regards,
> C. L. Martinez

Hi folks,

I run into exactly the same issue with my ldap proxy. When one node is master
for both carp nodes it responds to packets addressed to the carp ip. In load
balancing mode it sometimes responds depending on the source ip.

Please open a bug report.

-- 
wq: ~uw

Re: CARP load balancing problems under KVM

[Carlos Lopez]

Ok, done. I have already sent the bug report.

On 21/10/2020, 11:11, "Uwe Werler"  wrote:

On 21 Oct 07:12, Carlos Lopez wrote:
> Hi all,
> 
> Before upgrade from OpenBSD 6.7 to OpenBSD 6.8, my pair firewalls was 
using carp in IP balance mode without problems from several months. These 
firewalls are installed in a RHEL 8.2 (fully patched) KVM host.
> 
> After upgrading to OpenBSD 6.8, carp ip balance mode doesn’t works. I 
have tested reconfiguring balance mode for ip-stealth and ip-unicast also and 
the result is always the same: network packets are not processed by firewalls. 
But if I configure CARP using “the simple configuration” and one node is master 
and the other is backup all it is working without problems.
> 
> Since the configuration is exactly the same for version 6.8 as it was for 
version 6.7, could there be some kind of problem with the virtio driver on the 
network interfaces in version 6.8?
> 
> Regards,
> C. L. Martinez

Hi folks,

I run into exactly the same issue with my ldap proxy. When one node is 
master
for both carp nodes it responds to packets addressed to the carp ip. In load
balancing mode it sometimes responds depending on the source ip.

Please open a bug report.

-- 
wq: ~uw

Re: CARP with /30 ?

[Alarig Le Lay]

On 24/10/2019 10:41, Axel Rau wrote:
> Hi all,
> 
> does a CARP setup with 2 firewll boxes with an upstream /30 transfer net i 
> feasible?
> E.g.
> 
> 5.6.7.232/30
> 
> 5.6.7.232 if box1
> 5.6.7.233 upstream router
> 5.6.7.234 if box2
> 5.6.7.235 if CARP
> 
> Quick answer would be very helpfull.
> 
> Thanks, Axel
> ---
> PGP-Key: CDE74120  ☀  computing @ chaos claudius
> 

You can’t use 4 addresses on a /30, 5.6.7.232 is the net and 235 the
broadcast.
Anyway, there is use to put anything but a /32 on a VIP.

Also, why do you want CARP with an upstream? Just do BGP or ECMP.

-- 
Alarig

Re: CARP on Hyper-V VM

[Henry Bonath]

Were you able to resolve?

I have a SCVMM environment and I ran into a similar issue, there is a bug
in VMM 2016 with the NDIS extension but I believe its resolved in 1807.
I have not been able to test, but would like to know if you had any
success, as I was not able to use CARP at all in Hyper-V as you stated.

On Tue, Oct 16, 2018 at 6:41 AM Markus Rosjat  wrote:

> Hi Ricardo,
>
> > You must set the VM's network adapter to 'Enable MAC address spoofing'
> > under 'Advanced Features'.
>
> nope this isn't solving the problem. I can only ping the virtual ip from
> the local machine still. It might need the NDIS Extention enabled on the
> vSwitch too but I did't changed that because of the probable network
> disconnection. I will give it a shot later.
>
> regards
>
> MArkus
>
> --
> Markus Rosjatfon: +49 351 8107224mail: ros...@ghweb.de
>
> G+H Webservice GbR Gorzolla, Herrmann
> Königsbrücker Str. 70, 01099 Dresden
>
> http://www.ghweb.de
> fon: +49 351 8107220   fax: +49 351 8107227
>
> Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before
> you print it, think about your responsibility and commitment to the
> ENVIRONMENT
>
>

Re: CARP on Hyper-V VM

[Markus Rosjat]

Hi Ricardo,


You must set the VM's network adapter to 'Enable MAC address spoofing'
under 'Advanced Features'.


nope this isn't solving the problem. I can only ping the virtual ip from 
the local machine still. It might need the NDIS Extention enabled on the 
vSwitch too but I did't changed that because of the probable network 
disconnection. I will give it a shot later.


regards

MArkus

--
Markus Rosjatfon: +49 351 8107224mail: ros...@ghweb.de

G+H Webservice GbR Gorzolla, Herrmann
Königsbrücker Str. 70, 01099 Dresden

http://www.ghweb.de
fon: +49 351 8107220   fax: +49 351 8107227

Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before you 
print it, think about your responsibility and commitment to the ENVIRONMENT

Re: CARP on Hyper-V VM

[Ricardo Mestre]

Hi Markus,

You must set the VM's network adapter to 'Enable MAC address spoofing'
under 'Advanced Features'.

/mestre

On 10:03 Tue 16 Oct , Markus Rosjat wrote:
> Hi there,
> 
> i just have a question to CARP on Hyper-V VMs. It seems there was a
> problemwith the virtual IP not be reachable from somewere else then the
> machine itself. Since I try to set up CARP on such a VM an noticed the same
> behaviour on a OpenBSD 6.1 I wonder if this issue is resolved in 6.3?
> 
> regards
> 
> -- 
> Markus Rosjatfon: +49 351 8107224mail: ros...@ghweb.de
> 
> G+H Webservice GbR Gorzolla, Herrmann
> Königsbrücker Str. 70, 01099 Dresden
> 
> http://www.ghweb.de
> fon: +49 351 8107220   fax: +49 351 8107227
> 
> Bitte prüfen Sie, ob diese Mail wirklich ausgedruckt werden muss! Before you 
> print it, think about your responsibility and commitment to the ENVIRONMENT
> 

Re: carp ssh setup

[Kapetanakis Giannis]

On 17/04/18 02:06, jungle Boogie wrote:
> Hi All,
> 
> I have a very simple carp setup - basically I want ssh access if the
> master goes offline.
> In theory, this are functioning correctly. In practice, it seems the
> backup is taking over way too often - the backup takes over way too
> often, even when I'm ssh'd to the master device.
> 
> master:
> inet 192.168.0.99 255.255.255.0 192.168.0.255 vhid 1 carpdev dwxe0
> state master advskew 1 pass pass
> 
> backup:
> inet 192.168.0.99 255.255.255.0 192.168.0.255 vhid 1 carpdev dwxe0
> state backup advskew 10 pass pass

Do you have IP addresses on the physical interfaces dwxe0?

Use netmask 255.255.255.255 on carp devices.
and then use the IP address off the physical interface (dwxe0) to access each 
firewall.

ifconfig -A 
would be useful for more help

G 

Re: carp and squid

[Craig Skinner]

Hi Frank,

On Wed, 21 Dec 2016 12:41:43 +0100 Frank White wrote:
> Does 2 nodes clustered openbsd firewall work with squid?
> Is there any specific configuration?
> 

carp may not be needed as:
*) PAC files can list multiple proxies
*) A DNS entry can have multiple IP addresses

See the Squid FAQ:
http://wiki.squid-cache.org/SquidFaq/ConfiguringBrowsers#Redundant_Proxy_Auto-Configuration
http://wiki.squid-cache.org/SquidFaq/ConfiguringBrowsers#Fully_Automatic_Configuration

Also: http://FindProxyForURL.com/example-pac-file/

Symlink a proxy.pac file as wpad.dat

Cheers,
-- 
Craig Skinner | http://linkd.in/yGqkv7

Re: carp and squid

[Jiri B]

On Wed, Dec 21, 2016 at 12:41:43PM +0100, Frank White wrote:
> Hi, does 2 nodes clustered openbsd firewall work with squid ?
> is there any specific configuration ?

If squid on each node would have its own cache dir, ie. not sharing
data, then pointing your clients to squid hostname linked to CARP
IP should work, shouldn't it?

If squid daemons on both nodes would share cache dir, then you should
somehow prevent "failed" node not to continue to mess with storage.
Typical solution is STONITH (shoot the other node in the head - ie.
power fencing). Then you could maybe use ifstated to monitor CARP interface
and start squid daemon if CARP IP is local.

I would be also interested in solutions used by various
OpenBSD users.

j.

Re: CARP, BGP and OpenBSD routers - integration tips/suggestions needed

[Bob Jones]

Hi Bryan,

Thank you for the great message.  I will re-read it in more detail
over the next few days and have a go at getting all the pieces of the
jigsaw put together !

Thanks again.

On 11 December 2016 at 18:12, Bryan Vyhmeister  wrote:
> On Sun, Dec 11, 2016 at 09:45:08AM +, Bob Jones wrote:
>> I have a planned network topology that will run on OpenBSD that (at
>> the moment) will constitute of three boxes :
>>
>> 1 x Router (Openbsd running bgpd for connection to the outside world)
>> 2 x Firewalls (running Openbsd)
>>
>> I can't quite figure out the best way to deal with the "external" side
>> of the firewalls ? (Obviously the "internal" side would be CARP).
>
> The missing piece here is OSPF. The paper below is what I used as my
> template to setup my network which is very much like your design. I then
> added the CARP configuration which I will explain below.
>
> http://www.openbsd.org/papers/linuxtag06-network.pdf
>
>> At the moment, since the devices are located in the same rack, I am
>> thinking of running a patch cable directly from each firewall to two
>> ports on the Router (i.e. F1a -> R1a and F2a to R1b).  The reason for
>> this is to avoid going via a switch and adding a point of failure
>> (yes, I know, I only have one "router" but hopefully that will
>> change in the not too distant future !)
>
> I have an external router running BGP with my provider with four
> ethernet ports. The first interface (em0) is connected to my provider.
> The other ports (em1, em2, and em3) are all part of bridge0 which is
> what my other two routers are connected to. The internal IP address
> (which is .1 of a /29 and the beginning of my /24) of my external router
> exists on vether0 also added to bridge0. I run iBGP and OSPF between the
> three routers as in the paper above. That means the first internal
> router would have .3 on its em0 and the second internal router would
> have .4 on its em1. I am actually changing out to use a switch because
> once in a while the external router stops seeing OSPF from the internal
> routers. I can't be positive it has anything to do with bridge(4)
> because the routers have not been upgraded recently past 5.8 and there
> have been improvements to lots of areas. This week I am upgrading them
> to 6.0-stable and replacing some hardware. I have this same setup in a
> datacenter as well all running 6.0-stable that uses a switch and has
> worked perfectly for several years.
>
>> The problem is I can't quite figure out the OpenBSD software
>> configuration for that concept and how it inter-relate with CARP
>> running on the "internal" side of the firewalls ?  Should I be running
>> OSPF ? iBGP ?  Or something else (switchd ? vether ?)
>
> Like I mentioned above, you want BGP to your provider on the external
> router. On its internal interface use iBGP and OSPF to the other two
> routers. You can use vether(4) and bridge(4) on the external router's
> internal interfaces like I did which seems to work fairly well. On the
> two internal routers, the key point for CARP to work is to use the
> "demote carp" option as documented in ospfd.conf(5) and use the carp
> interface(s) rather than the physical in ospfd.conf on the internal side
> of the internal routers. You also have to have a link between the
> internal routers for pfsync(4) and an OSPF link. Here is a simplified
> snippet from my ospfd.conf on one of the internal routers.
>
> area 0.0.0.0 {
> demote carp
> interface em0 { metric 10 }
> interface em1 { metric 20 }
> interface carp2 { passive }
> }
>
> In this case, em0 connects to the external router. Interface em1 is a
> cable between the two internal routers which provides both a /30 link
> between them for OSPF and also pfsync for CARP to work correctly. I
> simplified my snippet above because I have some other things working
> that would complicate your setup. In a setup like you want, carp2 would
> correspond to em2 for example. I am using a /25 for the internal network
> using CARP. So carp2 has the .125 IP address shared bewteen both
> internal routers, em2 on the first internal router has .126, and em2 on
> the second internal router has .127 to allow CARP to work correctly.
>
> Hopefully this helps you get things going. This setup works very well
> for me at multiple sites and can easily be expanded by adding another
> external router to another provider in the future.
>
> Bryan

Re: CARP, BGP and OpenBSD routers - integration tips/suggestions needed

[Bryan Vyhmeister]

On Sun, Dec 11, 2016 at 09:45:08AM +, Bob Jones wrote:
> I have a planned network topology that will run on OpenBSD that (at
> the moment) will constitute of three boxes :
> 
> 1 x Router (Openbsd running bgpd for connection to the outside world)
> 2 x Firewalls (running Openbsd)
> 
> I can't quite figure out the best way to deal with the "external" side
> of the firewalls ? (Obviously the "internal" side would be CARP).

The missing piece here is OSPF. The paper below is what I used as my
template to setup my network which is very much like your design. I then
added the CARP configuration which I will explain below.

http://www.openbsd.org/papers/linuxtag06-network.pdf

> At the moment, since the devices are located in the same rack, I am
> thinking of running a patch cable directly from each firewall to two
> ports on the Router (i.e. F1a -> R1a and F2a to R1b).  The reason for
> this is to avoid going via a switch and adding a point of failure
> (yes, I know, I only have one "router" but hopefully that will
> change in the not too distant future !)

I have an external router running BGP with my provider with four
ethernet ports. The first interface (em0) is connected to my provider.
The other ports (em1, em2, and em3) are all part of bridge0 which is
what my other two routers are connected to. The internal IP address
(which is .1 of a /29 and the beginning of my /24) of my external router
exists on vether0 also added to bridge0. I run iBGP and OSPF between the
three routers as in the paper above. That means the first internal
router would have .3 on its em0 and the second internal router would
have .4 on its em1. I am actually changing out to use a switch because
once in a while the external router stops seeing OSPF from the internal
routers. I can't be positive it has anything to do with bridge(4)
because the routers have not been upgraded recently past 5.8 and there
have been improvements to lots of areas. This week I am upgrading them
to 6.0-stable and replacing some hardware. I have this same setup in a
datacenter as well all running 6.0-stable that uses a switch and has
worked perfectly for several years.

> The problem is I can't quite figure out the OpenBSD software
> configuration for that concept and how it inter-relate with CARP
> running on the "internal" side of the firewalls ?  Should I be running
> OSPF ? iBGP ?  Or something else (switchd ? vether ?)

Like I mentioned above, you want BGP to your provider on the external
router. On its internal interface use iBGP and OSPF to the other two
routers. You can use vether(4) and bridge(4) on the external router's
internal interfaces like I did which seems to work fairly well. On the
two internal routers, the key point for CARP to work is to use the
"demote carp" option as documented in ospfd.conf(5) and use the carp
interface(s) rather than the physical in ospfd.conf on the internal side
of the internal routers. You also have to have a link between the
internal routers for pfsync(4) and an OSPF link. Here is a simplified
snippet from my ospfd.conf on one of the internal routers.

area 0.0.0.0 {
demote carp
interface em0 { metric 10 }
interface em1 { metric 20 }
interface carp2 { passive }
}

In this case, em0 connects to the external router. Interface em1 is a
cable between the two internal routers which provides both a /30 link
between them for OSPF and also pfsync for CARP to work correctly. I
simplified my snippet above because I have some other things working
that would complicate your setup. In a setup like you want, carp2 would
correspond to em2 for example. I am using a /25 for the internal network
using CARP. So carp2 has the .125 IP address shared bewteen both
internal routers, em2 on the first internal router has .126, and em2 on
the second internal router has .127 to allow CARP to work correctly.

Hopefully this helps you get things going. This setup works very well
for me at multiple sites and can easily be expanded by adding another
external router to another provider in the future.

Bryan

Re: CARP host with lower advskew not becoming master

[Peter Hessler]

On 2016 Oct 04 (Tue) at 09:27:50 +0200 (+0200), Jasper Siepkes wrote:
:Hi list!
:
:I'm experimenting with CARP and I'm a bit puzzled by the following
:behavior; I have 2 hosts setup in an active/passive way with CARP. 
:Host A has an advskew of 0 and becomes master, Host B has an
:advskew of 100 and becomes backup. Now when host A fails host B becomes
:master just like i would expect. However once host A comes backup again
:he doesn't become master, he stays backup even though he has a 
:lower advertise skew. 
:

ifconfig -g carp

Compare the carp demote counters.  If they are different, the higher one
refuses to take master.


-- 
For 20 dollars, I'll give you a good fortune next time ...

Re: CARP host with lower advskew not becoming master

[Jasper Siepkes]

Silly me... I forgot the 'net.inet.carp.preempt' sysctl variable.

I thought it was only for forcing demotion of other CARP interfaces if a 
single one failed. But it's also for "claiming" the master spot.

Sorry for the noise :-(

> Op 4 oktober 2016 om 9:27 schreef Jasper Siepkes :
> 
> Hi list!
> 
> I'm experimenting with CARP and I'm a bit puzzled by the following
> behavior; I have 2 hosts setup in an active/passive way with CARP. 
> Host A has an advskew of 0 and becomes master, Host B has an
> advskew of 100 and becomes backup. Now when host A fails host B becomes
> master just like i would expect. However once host A comes backup again
> he doesn't become master, he stays backup even though he has a 
> lower advertise skew. 
> 
> Peeking with tcpdump tells me host A just goes to backup and doesn't
> advertise at all so host B never knows a host with lower advskew 
> came up.
> 
> That's not what I expected. Is that normal? From all the examples I 
> can find on the net I would expect host A to become master again. For
> example a lot of 'ifstated' examples use the advskew to promote or
> demote a host as master but since a host with lower advskew doesn't 
> seem to 'claim' the master position those examples don't work.
> 
> The setup is a cleanly installed OpenBSD 6.0 with the only 
> modifications the configs below. I've tested this in a VM and on 
> baremetal.
> 
> Host A
> 
> hostname.em1:
> 
> inet 10.253.255.2 255.255.254.0 NONE
> 
> 
> hostname.carp1000:
> 
> 
> carpdev em1 advbase 1 advskew 0 pass foo vhid 20
> inet 10.253.255.1 255.255.254.0 NONE
> carppeer 10.253.255.3
> 
> 
> Host B
> 
> hostname.em1:
> 
> inet 10.253.255.3 255.255.254.0 NONE
> 
> 
> hostname.carp1000:
> 
> carpdev em1 advbase 1 advskew 100 pass foo vhid 20
> inet 10.253.255.1 255.255.254.0 NONE
> carppeer 10.253.255.2
> 
> 
> Kind regards,
> 
> Jasper

Re: Carp and VLANs

[Andrew Seguin]

Thank you,

This (having unique VHID) was the solution.

I had considered originally that since each carp device is on its own VLAN,
that would represent a unique broadcast domain and it wouldn't be violating
anything - but without your suggestion I'm not sure I would have gone back
to review that decision.

I'm still a bit curious how it came to that. I did snoop if carp
announcements were leaking from a tagged vlan onto the default network, but
didn't see any sign of that. So maybe it was because the VLANs were riding
on top of the same physical interface... but a lot less important now.

Regards,
Andrew




On Tue, Aug 23, 2016 at 8:34 PM, John Jasen 
wrote:

> All your carp devices have the same VHID. As two share the same network,
> that could cause problems.
>
>
>
>
> On 08/23/2016 01:40 PM, Andrew Seguin wrote:
> > Hi,
> >
> > I'm building up an OpenBSD router/firewall (migrating away from FreeBSD)
> > but have been blocked by a behavior of carp in combination with VLANs
> that
> > I didn't expect or experience before. I'm hoping somebody could enlighten
> > me a little bit about why carp floating IPs stop working when the carp
> > status is master for the physical interface.
> >
> >
> > Originally, there was a pair of FreeBSD systems (FW1 and FW2) where I had
> > no issues with carp managed IPs.
> >
> > At the moment, one system is reinstalled with OpenBSD 5.9 (FW1), the
> other
> > remains with FreeBSD (FW2).
> >
> > The network is setup in such a way that the default vlan (1) is untagged,
> > and this network is for all the network management. All other traffic
> goes
> > over tagged networks. The network switches we have simply work in this
> way
> > and so I can't make vlan 1 also a tagged interface to test the impact of
> > such a configuration.
> >
> > As long as the OpenBSD system is not the master for the default /
> untagged
> > network associated to the physical network interface, the system will
> > accept packets for its CARP IPs.
> >
> > When OpenBSD becomes master for the untagged network, it won't forward or
> > respond (ping) to packets addressed to its floating IP.
> >
> > Configuration files for the physical interface (sk0) and a couple VLANs
> (I
> > run a dozen, but trimmed back to two for the purpose of this mail).
> >
> > # cat /etc/sysctl.conf
> > net.inet.carp.allow=1
> > net.inet.carp.preempt=1
> > net.inet.ip.forwarding=1
> >
> > # cat /etc/hostname.sk0
> >   inet
> > 10.1.0.2 255.255.255.0 NONE description "main link"
> > inet 10.0.0.2 255.255.255.0
> >
> > # cat /etc/hostname.carp1
> > vhid 1 pass password carpdev sk0 advskew 150
> > inet 10.1.0.1 255.255.255.0
> > inet alias 10.0.0.1 255.255.255.0
> >
> > # cat /etc/hostname.vlan10
> > inet 10.10.0.2 255.255.255.0 NONE vlan 10 vlandev sk0 description
> "Printer
> > network"
> >
> > # cat /etc/hostname.carp10
> > vhid 1
> pass
> > password carpdev vlan10 advskew 150
> > inet 10.10.0.1 255.255.255.0
> >
> > # cat /etc/hostname.vlan50
> > inet 10.50.0.2 255.255.255.0 NONE vlan 50 vlandev sk0 description
> "Wireless
> > backbone"
> >
> > # cat /etc/hostname.carp50
> > vhid 1 pass password carpdev vlan50 advskew 150
> > inet 10.50.0.1 255.255.255.0
> >
> >
> > The other system has a similar configuration with the exception that IPs
> > ending in .2 are .3 on FW2 and FW2 has advskew 100.
> >
> >
> > If I make FW1 (OpenBSD) the master for vlan10 and vlan50 (ifconfig carp10
> > advskew 1; ifconfig carp50 advskew) but not for sk0, then it will forward
> > packets between those two networks without problem and ping 10.10.0.1
> works
> > fine.
> >
> > The moment I make it the master for sk0 (ifconfig carp1 advskew 1), it no
> > longer forwards packets (between vlan10 and vlan50, vlan10 and the
> untagged
> > vlan) and it no longer responds to ping for any of the IPs associated to
> > the carp interfaces from external systems (ping 10.10.0.2 works, ping
> > 10.10.0.1 doesn't work) although from the local box it works (ping
> > 10.10.0.1 from FW1 works). Output from ifconfig shows FW1 is the master
> for
> > all interfaces.
> >
> > Throughout, I am able to keep working with the box remotely as long as I
> > logged in via the local subnet IP (ie: from a workstation with IP
> > 10.10.0.50, I can ssh to 10.10.0.2).
> >
> > For testing ... while the FW1 (OpenBSD) is master for all interfaces, I
> > used tcpdump and could see the packets arriving at the system only if I
> > took the dump on sk0 or carp1. No packets show up on vlan10 or carp10 for
> > the box. On vlan10 - I can see all traffic addressed to 10.10.0.2 without
> > problem. On carp10 - I only see the "CARPv2-advertise" and arp
> > request/response packets.
> >
> > To rule things out, I've kept the PF configuration as simple as possible
> > for testing (simply 1 line: "pass").
> >
> > I always made sure that the 

Re: Carp and VLANs

[John Jasen]

All your carp devices have the same VHID. As two share the same network,
that could cause problems.




On 08/23/2016 01:40 PM, Andrew Seguin wrote:
> Hi,
>
> I'm building up an OpenBSD router/firewall (migrating away from FreeBSD)
> but have been blocked by a behavior of carp in combination with VLANs that
> I didn't expect or experience before. I'm hoping somebody could enlighten
> me a little bit about why carp floating IPs stop working when the carp
> status is master for the physical interface.
>
>
> Originally, there was a pair of FreeBSD systems (FW1 and FW2) where I had
> no issues with carp managed IPs.
>
> At the moment, one system is reinstalled with OpenBSD 5.9 (FW1), the other
> remains with FreeBSD (FW2).
>
> The network is setup in such a way that the default vlan (1) is untagged,
> and this network is for all the network management. All other traffic goes
> over tagged networks. The network switches we have simply work in this way
> and so I can't make vlan 1 also a tagged interface to test the impact of
> such a configuration.
>
> As long as the OpenBSD system is not the master for the default / untagged
> network associated to the physical network interface, the system will
> accept packets for its CARP IPs.
>
> When OpenBSD becomes master for the untagged network, it won't forward or
> respond (ping) to packets addressed to its floating IP.
>
> Configuration files for the physical interface (sk0) and a couple VLANs (I
> run a dozen, but trimmed back to two for the purpose of this mail).
>
> # cat /etc/sysctl.conf
> net.inet.carp.allow=1
> net.inet.carp.preempt=1
> net.inet.ip.forwarding=1
>
> # cat /etc/hostname.sk0
>   inet
> 10.1.0.2 255.255.255.0 NONE description "main link"
> inet 10.0.0.2 255.255.255.0
>
> # cat /etc/hostname.carp1
> vhid 1 pass password carpdev sk0 advskew 150
> inet 10.1.0.1 255.255.255.0
> inet alias 10.0.0.1 255.255.255.0
>
> # cat /etc/hostname.vlan10
> inet 10.10.0.2 255.255.255.0 NONE vlan 10 vlandev sk0 description "Printer
> network"
>
> # cat /etc/hostname.carp10
> vhid 1 pass
> password carpdev vlan10 advskew 150
> inet 10.10.0.1 255.255.255.0
>
> # cat /etc/hostname.vlan50
> inet 10.50.0.2 255.255.255.0 NONE vlan 50 vlandev sk0 description "Wireless
> backbone"
>
> # cat /etc/hostname.carp50
> vhid 1 pass password carpdev vlan50 advskew 150
> inet 10.50.0.1 255.255.255.0
>
>
> The other system has a similar configuration with the exception that IPs
> ending in .2 are .3 on FW2 and FW2 has advskew 100.
>
>
> If I make FW1 (OpenBSD) the master for vlan10 and vlan50 (ifconfig carp10
> advskew 1; ifconfig carp50 advskew) but not for sk0, then it will forward
> packets between those two networks without problem and ping 10.10.0.1 works
> fine.
>
> The moment I make it the master for sk0 (ifconfig carp1 advskew 1), it no
> longer forwards packets (between vlan10 and vlan50, vlan10 and the untagged
> vlan) and it no longer responds to ping for any of the IPs associated to
> the carp interfaces from external systems (ping 10.10.0.2 works, ping
> 10.10.0.1 doesn't work) although from the local box it works (ping
> 10.10.0.1 from FW1 works). Output from ifconfig shows FW1 is the master for
> all interfaces.
>
> Throughout, I am able to keep working with the box remotely as long as I
> logged in via the local subnet IP (ie: from a workstation with IP
> 10.10.0.50, I can ssh to 10.10.0.2).
>
> For testing ... while the FW1 (OpenBSD) is master for all interfaces, I
> used tcpdump and could see the packets arriving at the system only if I
> took the dump on sk0 or carp1. No packets show up on vlan10 or carp10 for
> the box. On vlan10 - I can see all traffic addressed to 10.10.0.2 without
> problem. On carp10 - I only see the "CARPv2-advertise" and arp
> request/response packets.
>
> To rule things out, I've kept the PF configuration as simple as possible
> for testing (simply 1 line: "pass").
>
> I always made sure that the corresponding CARP interfaces were in a backup
> state on FW2 (freebsd) and via tcpdump that packets weren't ending up there
> by some accident of the switches.
>
> I've tried setting the subnet masks for the floating (carp) IP addresses to
> be 255.255.255.255 - didn't change the behavior.
>
> I set net.inet.carp.log=7 - nothing is noted in /var/log/messages beyond
> the transitions (carp1: state transition: BACKUP -> MASTER; MASTER ->
> BACKUP).
>
> Since then, I'm out of ideas what to try and am turning to the mailing list
> for help.
>
> I'm rather new to OpenBSD, but I reviewed the FAQ and searched on google,
> read man pages for carp, ifconfig, hostname.if, etc but didn't get any new
> ideas.
>
> Any ideas or suggestions what else I might look at?
>
> Is this expected behavior or have I overlooked some configuration option?
>
> Thanks in advance,
> Andrew

Re: Carp interface sitting on vlan can not be pinged

[Sebastian Benoit]

Kim Zeitler(kim.zeit...@konzept-is.de) on 2016.04.15 11:41:07 +0200:
> Hello
> 
> maybe a stupid question, but is it possible to run a carp(4) interface 
> on vlan(4) interfaces?

yes
 
> In the following setup we have the problem that both boxes can be pinged 
> on their address associated with their respective vlan(4) interface, but 
> not on the carp(4) interface IP. Both boxes are recent installs and are 
> running -current
> 
> em2 (no ip) ---> vlan100 (192.168.150.200) ---> carp2 (192.168.150.1)
> \
>  --> vlan101 (192.168.151.200) ---> carp3 (192.168.151.1)
> 
> respectively the corresponding node using .202 instead of .200 for the 
> vlan(4) interfaces

you did not send the output of

ifconfig vlan
ifconfig carp

this might help
 
> == The configuration ==
> 
> # uname -a
> OpenBSD router12 5.9 GENERIC.MP#1983 amd64
> 
> # cat /etc/hostname.em2
> up
> 
> # cat /etc/hostname.vlan100
> inet 192.168.150.200 255.255.255.0 192.168.150.255 vlan 100 vlandev em2

try to write this as

inet 192.168.150.200 255.255.255.0 NONE
vlan 100 vlandev em2

> # cat /etc/hostname.carp2
> inet 192.168.150.1 255.255.255.0 192.168.150.255 vhid 201 carpdev 
> vlan100 pass 1234 group wlan

inet 192.168.150.1 255.255.255.0 NONE
vhid 201 carpdev vlan100 pass 1234
group wlan

> # cat /etc/pf.conf

if above does not work, try pfctl -d
also, the pf.conf you show is not complete, so ...

> pass quick on {em2,vlan100,vlan101} proto carp
> ...
> pass inet proto icmp icmp-type $icmp_types
> pass vlan100:network
> ...
> 
> # netstat -rn
> ...
> 192.168.150/24 192.168.150.200UCP0 4401 - 4 
> vlan100
> 192.168.150/24 192.168.150.1  CP 00 - 4 
> carp2
> 192.168.150.1  00:00:5e:00:01:c9  UHLl   0 9981 - 1 
> carp2
> 192.168.150.20090:e2:ba:c1:11:11  UHLl   0   30 - 1 
> vlan100
> 192.168.150.255192.168.150.200UHPb   0   80 - 1 
> vlan100
> 192.168.150.255192.168.150.1  HPb00 - 1 
> carp2
> 192.168.151/24 192.168.151.200UCP1 3040 - 4 
> vlan101
> 192.168.151/24 192.168.151.1  CP 00 - 4 
> carp3
> 192.168.151.1  00:00:5e:00:01:ca  UHLl   0  182 - 1 
> carp3
> 192.168.151.20090:e2:ba:c1:11:11  UHLl   0   36 - 1 
> vlan101
> 192.168.151.255192.168.151.200UHPb   00 - 1 
> vlan101
> 192.168.151.255192.168.151.1  HPb00 - 1 
> carp3
> 
> 
> Cheers
> Kim
> 

-- 

Re: carp dhclient

[Jack J. Woehr]

Josh Grosse wrote:

On 2016-02-01 11:32, sven falempin wrote:

Dear Readers,
Without IP carp is marked as inactive,


See https://sites.google.com/site/bsdstuff/dhcarp and adapt
to your requirements.


The Book of PF, 3rd Edition
A No-Nonsense Guide to the OpenBSD Firewall
by Peter N. M. Hansteen
ISBN-10: 1-59327-589-7
ISBN-13: 978-1-59327-589-1
Copyright 2015.


--
Jack J. Woehr # Science is more than a body of knowledge. It's a way of
www.well.com/~jax # thinking, a way of skeptically interrogating the universe
www.softwoehr.com # with a fine understanding of human fallibility. - Carl Sagan

Re: carp dhclient

[Josh Grosse]

On 2016-02-01 11:32, sven falempin wrote:

Dear Readers,
Without IP carp is marked as inactive,
i tried to set up a stupid IP on it and then call dhclient.
It sends packet but does configure interface.
:'(
Any particular reason for this ?
Thank you.


Carp requires static addresses. You can establish carp on
an inward network, and use carp advertisements in concert
with ifstated(8) to transfer the use of a MAC address between
active systems on the outer, dynamic network.

See https://sites.google.com/site/bsdstuff/dhcarp and adapt
to your requirements.

Re: carp and bridge

[sven falempin]

On Tue, Jan 26, 2016 at 6:29 PM, sven falempin 
wrote:

> Dear readers,
>
> How bridge and carp interfaces works together ?
>
> can i bridge an interface that is a carpdev ?
> or should i bridge the carpdev ??
> will the different physical be advertise and
> would be able to contact the carp interface address ?
>
> Thank you for any input that would save a bit of try and error :-)
>
> --
>


For those who cares, you cannot carp a bridged interface but
you can bridge a carp ed interface. Trace and all reported.

OTHA, preempt seems to be activated by default , my sysctl is marked as =0
when i unplug the Master, slave is doing the job,
when i plug Master again it took back control, which would the preempt=1
behavior.

<<
For most scenarios it is desirable to have a well-defined master, achieved
by enabling the preempt option. Enable it on both host A and B:

# sysctl net.inet.carp.preempt=1
>>

This is cool, but if master reboot each minute ... and i want to keep the
switching fast preempt=0 is usefull,
switch is performed, alert is send, fix are made and when master is stable
it s activated again.

Cheers.



-- 
-
() ascii ribbon campaign - against html e-mail
/\

Re: carp/pfsync-problem: carp states stuck in "INIT" on boot on both machines but work correctly if called manually via /etc/netstart

[Andre Ruppert]

...I don't believe it...

I ssh'd all the time to the gateways and never had a look to the 
bootmessages


2x "ifconfig  invalid argument" was the hint at boot.

The fault (syntax typo?) was included in hostname.carp[0,1] -
"\" for a 2-liner didn't work... despite the usage of blanks only.

Crunched it to a 1-liner and all worked...

Seems that the parsing is different at booting?

Andre

Am 02.10.15 um 10:37 schrieb Andre Ruppert:

Hello @list,

perhaps I'm stupid but I've got a problem with two CARPed gateways
running  5.7-amd64 stable.

Hardware:
two supermicro-board machines with four network interfaces each (em0 ..
em3).

Networks:
LAN A : 172.16.210/24 via em0
LAN B : 172.16.0/24   via em1
direct connect for pfsync: 1.1.1.0/30 via em3


Gateway A setup --- (master) ---

hostname.em0:
"inet 172.16.210.2 255.255.255.0"

hostname.em1:
"inet 172.16.0.30 255.255.255.0"

hostname.em3
"inet 1.1.1.1 255.255.255.252 1.1.1.3"

hostname.carp0
"inet 172.16.210.1 255.255.255.0 172.16.210.255 vhid 1 \
   carpdev em0 pass gwvoip01carppass advskew 0"

hostname.carp1
"inet 172.16.0.29  255.255.255.0 172.16.0.255 vhid 2 \
   carpdev em1 pass gwvoip01carppass advskew 0"

hostname.pfsync0
"up syncdev em3 syncpeer 1.1.1.2"

sysctl net.inet.carp ->
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=7  #debugging


Gateway B setup --- (backup) ---

hostname.em0:
"inet 172.16.210.3 255.255.255.0"

hostname.em1:
"inet 172.16.0.31 255.255.255.0"

hostname.em3
"inet 1.1.1.2 255.255.255.252 1.1.1.3"

hostname.carp0
"inet 172.16.210.1 255.255.255.0 172.16.210.255 vhid 1 \
carpdev em0 pass gwvoip01carppass advskew 100"

hostname.carp1
"inet 172.16.0.29 255.255.255.0 172.16.0.255 vhid 2 \
carpdev em1 pass gwvoip01carppass advskew 100"

hostname.pfsync0
"up syncdev em3 syncpeer 1.1.1.1"

sysctl net.inet.carp ->
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=2


problem description --

(remark: failover-switching works on both machines in both directions)

If one of the machines reboots, the local carp-interfaces stuck in
"INIT" state. Same behavior on both systems.

The log (A) after reboot:
carp: carp0 demoted group carp by -1 to 162 (carpdev)
carp: carp1 demoted group carp by -1 to 161 (carpdev)
carp: pfsync0 demoted group carp by -1 to 32 (pfsync bulk done)
carp: pfsync0 demoted group pfsync by -1 to 32 (pfsync bulk done)
carp: pfsync0 demoted group carp by -32 to 0 (pfsync init)
carp: pfsync0 demoted group pfsync by -32 to 0 (pfsync init)

If the carp-interfaces are subsequently restartet via netstart command,
all works like a charm again...

The log (A) after "sh /etc/netstart [carp0,carp1]":
carp0: state transition: INIT -> BACKUP
state transition: BACKUP -> MASTER
state transition: INIT -> BACKUP
state transition: BACKUP -> MASTER

No PF-ruleset-problem!


resulting question -

what the heck is going on here? ;-)
alternative: what did I forgot to configure?

Thanks for reading...


Andre Ruppert

Re: carp(4) requires carpdev in OpenBSD 5.7

[Sebastian Benoit]

Rolf Sommerhalder(rolf.sommerhal...@alumni.ethz.ch) on 2015.08.01 17:17:42 
+0200:
 After upgrading a firewall cluster from 5.6 to 5.7, I observed that
 carpX interfaces failed to come up with their settings.
 
 A manual start 'sh /etc/netstart carpX' ran without errors, although
 carpX still did not get its settings from hostname.carpX.
 However, 'ifconfig carpX 10.0.8.1 vhid 108' failed:
  ifconfig: SIOCAIFADDR: Invalid argument
 
 http://www.openbsd.org/57.html revealed that carp(4) now needs to be
 configured with an explicit carpdev parent interface (as from 5.7).
 Thus adding 'carpdev vlanX' to the ifconfig or hostname.carpX solved
 my problem :-)
 
 Apparently, I had been mentioned in a previous revision of following
 -current, too.
 May I suggest to mention it in the Other changes section of the
 Upgrade 5.6-5.7 document?

maybe, i'll see.

 Also, it might be helpful to add carpdev to the examples, which is now
 mandatory, in the carp(4) man pages.

i just commited an update to that effect.

Thanks for your report.

Re: carp over vlan on trunk

[Stuart Henderson]

On 2015-03-14, pixelfairy pixelfa...@gmail.com wrote:
 OpenBSD r0 5.6 GENERIC#0 i386
 soekris net6501, dmesg below

 r0:/etc# cat hostname.trunk1
 trunkproto failover trunkport em4 trunkport em5
 up
 r0:/etc# cat hostname.vlan111
 inet 10.1.11.2 255.255.255.0 10.1.11.255 vlandev trunk1
 up
 r0:/etc# cat hostname.carp111
 inet 10.1.11.1 255.255.255.0 10.1.11.255 vhid 111 carpdev vlan111 pass ***
 r0:/etc# ifconfig carp111 10.1.11.1 netmask 255.255.255.0 vhid 111
 carpdev vlan111 pass ***
 ifconfig: SIOCAIFADDR: Can't assign requested address

 the trunk and vlans work fine. its the carp part thats not. is this
 possible? if so, how? ive tried google, but my search skills are weak.

Set carpdev/vhid/pass before you set the address.

# cat /etc/hostname.carp165
vhid 165 pass lalala carpdev vlan165
group office
inet 172.24.12.1/23

Re : CARP problem

[Mika]

Did you check layer 2 connectivity it seems the secondary firewall do not
receive any carp pack et

Mike

 Message original 
Objet : CARP problem
De : Jeff
À : misc@openbsd.org
Cc :

  I've been using CARP for years and it's always done exactly what I
  wanted and
  expected. We recently added a second ISP and another NIC to each of
  our
  firewalls. Each firewall now has 3 NIC's and three CARP interfaces.
  The
  original two are working fine, but the third CARP interface (carp2)
  shows up
  as MASTER on both the primary and failover firewalls. I have verified
  password,
  vhid and pf.conf and still can't figure out what I might have done
  wrong.

  Both firewalls have net.inet.carp.preempt=1

  Here is some output from tcpdump:

  firewall-master

  10:34:01.697488 CARPv2-advertise 36: vhid=10 advbase=1 advskew=15
  demote=0 (DF) [tos 0x10]
  10:34:01.975823 CARPv2-advertise 36: vhid=10 advbase=1 advskew=99
  demote=0 (DF) [tos 0x10]
  10:34:02.767475 CARPv2-advertise 36: vhid=10 advbase=1 advskew=15
  demote=0 (DF) [tos 0x10]
  10:34:03.375808 CARPv2-advertise 36: vhid=10 advbase=1 advskew=99
  demote=0 (DF) [tos 0x10]
  10:34:03.837465 CARPv2-advertise 36: vhid=10 advbase=1 advskew=15
  demote=0 (DF) [tos 0x10]
  10:34:04.776092 CARPv2-advertise 36: vhid=10 advbase=1 advskew=99
  demote=0 (DF) [tos 0x10]
  10:34:04.907466 CARPv2-advertise 36: vhid=10 advbase=1 advskew=15
  demote=0 (DF) [tos 0x10]
  10:34:05.977465 CARPv2-advertise 36: vhid=10 advbase=1 advskew=15
  demote=0 (DF) [tos 0x10]
  10:34:06.176254 CARPv2-advertise 36: vhid=10 advbase=1 advskew=99
  demote=0 (DF) [tos 0x10]

  firewall-backup

  10:34:42.225616 CARPv2-advertise 36: vhid=10 advbase=1 advskew=15
  demote=0 (DF) [tos 0x10]
  10:34:42.449469 CARPv2-advertise 36: vhid=10 advbase=1 advskew=99
  demote=0 (DF) [tos 0x10]
  10:34:43.295464 CARPv2-advertise 36: vhid=10 advbase=1 advskew=15
  demote=0 (DF) [tos 0x10]
  10:34:43.849458 CARPv2-advertise 36: vhid=10 advbase=1 advskew=99
  demote=0 (DF) [tos 0x10]
  10:34:44.365459 CARPv2-advertise 36: vhid=10 advbase=1 advskew=15
  demote=0 (DF) [tos 0x10]
  10:34:45.249484 CARPv2-advertise 36: vhid=10 advbase=1 advskew=99
  demote=0 (DF) [tos 0x10]
  10:34:45.435175 CARPv2-advertise 36: vhid=10 advbase=1 advskew=15
  demote=0 (DF) [tos 0x10]

  Suggestions please?

  Thanks!

  --

Re: carp failover problem

[Leclerc, Sebastien]

  Will try it during the weekend...
 

After reconnecting the firewalls differently, I got it fixed.
Logically, the connections are the same, but apparently the 5300xl had a hard 
time with its arp table...
Instead of connecting both firewalls directly on the routing switch, I made a 
trunk back to the 2524, and connected the firewalls there.
Within seconds after disconnecting a port or rebooting either firewall, carp 
now handles the failover smoothly!

Thanks!

Sebastien

Re: carp failover problem

[Leclerc, Sebastien]

 Rebooted fw2 at 3h02, fw1 kept master state, but had downtime until 3h12
 Rebooted fw1 at 3h15, got downtime until 4h10, fw1 got master state at 3h16, 
 fw2 got backup state at the same time
 

Inspecting further my logs, I see that smtp services were functioning between 
wan and dmz during the downtime period.  Our monitoring is done from the lan, 
so I suspect the 5300xl is causing the problem...
Any thoughts?

Thanks

Sebastien

Re: carp failover problem

[Leclerc, Sebastien]

Jan 30, 2015; 8:10am Stuart Henderson wrote :

/etc/hostname.carp0
advskew 0 carpdev em0 carppeer 192.168.3.10 pass secret1 state master
vhid 1 inet 192.0.2.2/28

Maybe unrelated, but it's not usual to set state master like this.

I know, it was not in the config at first, I added it to test.

Also inet should normally be at the start of a line in hostname.if.

Fails miserably if I do it :(
Only aliases get assigned to the interface, and a message indicates that the 
address cannot be assigned to the interface (I don't have the exact message, I 
rebooted after the failure, and it's not in the logs...)

My config was like this :

inet 192.0.2.2/28
advskew 0 carpdev em0 pass secret1 state master vhid 1
alias 192.0.2.3/32

I also tried with this, with the same result :

inet 192.0.2.2/28 advskew 0 carpdev em0 pass secret1 state master vhid 1
alias 192.0.2.3/32

Do things work if you use the default multicast, rather than carppeer?

As you can see above, I removed the carppeer from the config.
I had to add back the addresses manually to the carp interfaces, but then I got 
worst results : fw1 was master on all carp interfaces, but fw2 was backup on 
carp0 and carp2, and master on carp1
So I reverted to my previous configuration.

I changed some pf rules yesterday (removed antispoof) and disabled sasyncd, and 
rebooted during the night.
At least in the morning, everything was ok, but inspecting our monitoring 
system, here is what I found :

Rebooted fw2 at 3h02, fw1 kept master state, but had downtime until 3h12
Rebooted fw1 at 3h15, got downtime until 4h10, fw1 got master state at 3h16, 
fw2 got backup state at the same time

Thanks for your help


This mail was missing a few things. dmesg and ifconfig -A output would
be useful for starters (then we don't have to wonder how netstart parsed
your files).

Fw1 :

lo0: flags=8049UP,LOOPBACK,RUNNING,MULTICAST mtu 33144
priority: 0
groups: lo
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
inet 127.0.0.1 netmask 0xff00
em0: flags=8b43UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST mtu 
1500
lladdr 00:25:90:f2:6e:9a
priority: 0
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 192.168.3.9 netmask 0xfffc broadcast 192.168.3.11
inet6 fe80::225:90ff:fef2:6e9a%em0 prefixlen 64 scopeid 0x1
em1: flags=8b43UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST mtu 
1500
lladdr 00:25:90:f2:6e:9b
priority: 0
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 192.168.3.1 netmask 0xfff8 broadcast 192.168.3.7
inet6 fe80::225:90ff:fef2:6e9b%em1 prefixlen 64 scopeid 0x2
em2: flags=8b43UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST mtu 
1500
lladdr 00:25:90:f2:6e:9c
priority: 0
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 192.168.3.13 netmask 0xfffc broadcast 192.168.3.15
inet6 fe80::225:90ff:fef2:6e9c%em2 prefixlen 64 scopeid 0x3
em3: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500
lladdr 00:25:90:f2:6e:9d
priority: 0
media: Ethernet autoselect (1000baseT 
full-duplex,master,rxpause,txpause)
status: active
inet 192.168.3.17 netmask 0xfffc broadcast 192.168.3.19
inet6 fe80::225:90ff:fef2:6e9d%em3 prefixlen 64 scopeid 0x4
enc0: flags=41UP,RUNNING
priority: 0
groups: enc
status: active
tun0: flags=8051UP,POINTOPOINT,RUNNING,MULTICAST mtu 1500
priority: 0
groups: tun
status: active
inet 10.233.0.1 -- 10.233.0.2 netmask 0x
pfsync0: flags=41UP,RUNNING mtu 1500
priority: 0
pfsync: syncdev: em3 syncpeer: 192.168.3.18 maxupd: 128 defer: off
groups: carp pfsync
pflog0: flags=141UP,RUNNING,PROMISC mtu 33144
priority: 0
groups: pflog
carp0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500
lladdr 00:00:5e:00:01:01
priority: 0
carp: MASTER carpdev em0 vhid 1 advbase 1 advskew 0 carppeer 
192.168.3.10
groups: carp egress
status: master
inet6 fe80::200:5eff:fe00:101%carp0 prefixlen 64 scopeid 0x7
inet 192.0.2.2 netmask 0xfff0 broadcast 192.0.2.15
inet 192.0.2.3 netmask 0x
inet 192.0.2.4 netmask 0x
inet 192.0.2.5 netmask 0x
carp1: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500
lladdr 00:00:5e:00:01:02
priority: 0
carp: MASTER carpdev em1 vhid 2 advbase 1 advskew 0 carppeer 192.168.3.4
groups: carp
status: master
inet6 fe80::200:5eff:fe00:102%carp1 prefixlen 64 scopeid 0x8
inet 192.168.3.6 netmask 0x
carp2: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500
lladdr 00:00:5e:00:01:03
priority: 0
carp: MASTER carpdev 

Re: carp failover problem

[Christopher Barry]

On Fri, 30 Jan 2015 17:18:07 -0500
Leclerc, Sebastien sebastien.lecl...@saint-georges.ca wrote:

 Rebooted fw2 at 3h02, fw1 kept master state, but had downtime until
 3h12 Rebooted fw1 at 3h15, got downtime until 4h10, fw1 got master
 state at 3h16, fw2 got backup state at the same time
 

Inspecting further my logs, I see that smtp services were functioning
between wan and dmz during the downtime period.  Our monitoring is
done from the lan, so I suspect the 5300xl is causing the problem...
Any thoughts?

Thanks

Sebastien


the issue I had with Procurve switches was related to it's STP
implementation. strange things were happening while trying to PXE
boot a large number of Linux cluster nodes using gpxe. Swapping out the
switch with a different brand solved the problem, and I never revisited
it.

if you can do a quick test on a different switch, that would at least
rule that out as your issue. if not, try disabling STP and retest.

-C

Re: carp failover problem

[Leclerc, Sebastien]

if you can do a quick test on a different switch, that would at least

rule that out as your issue. if not, try disabling STP and retest


That was my guess, using a trunk to link the vlan to an edge switch not 
affected by stp, and connecting the firewalls there.
This way, the 5300xl won't have to detect which port is connected to the 
gateway (the 5300xl is a routing switch for the lan)
Will try it during the weekend...

Sebastien

Re: carp failover problem

[Stuart Henderson]

On 2015-01-27, Christopher Barry christopher.r.ba...@gmail.com wrote:
 On Tue, 27 Jan 2015 12:01:37 -0500
 Leclerc, Sebastien sebastien.lecl...@saint-georges.ca wrote:

/etc/hostname.carp0
advskew 0 carpdev em0 carppeer 192.168.3.10 pass secret1 state master
vhid 1 inet 192.0.2.2/28

Maybe unrelated, but it's not usual to set state master like this.
Also inet should normally be at the start of a line in hostname.if.

Do things work if you use the default multicast, rather than carppeer?

This mail was missing a few things. dmesg and ifconfig -A output would
be useful for starters (then we don't have to wonder how netstart parsed
your files).

 Well, it's been many years since I ran carp, so I cannot actually help
 with the carp config, but I can absolutely say that I have experienced a
 lot of unexplainable weirdness with ProCurve switches, so I can
 appreciate your suspicions there. I'll never buy another.

Procurve switches have been working nicely for me in various setups
involving carp etc. I've used various: 2626 2824 2510-24 4200vl 5300zl
2530-24g etc. Not saying it's impossible but other areas seem more likely.

Re: carp failover problem

[Christopher Barry]

On Tue, 27 Jan 2015 12:01:37 -0500
Leclerc, Sebastien sebastien.lecl...@saint-georges.ca wrote:

Hi,

I have two firewalls in a carp failover setup, but the failover does
not work as expected... The problem happens when I reboot the backup
firewall (while in backup state). Just after the reboot, I have these
entries in dmesg :

carp0: state transition: BACKUP - MASTER
carp1: state transition: BACKUP - MASTER
carp0: state transition: MASTER - BACKUP
carp1: state transition: MASTER - BACKUP

Why would there be no mention of carp2?
And no corresponding entries on the master?

States are consistent (all backup on backup, and all master on
master), but forwarded connections hang, until I force back the master
with this :
 sudo ifconfig -g carp carpdemote 128
 sudo ifconfig -g carp -carpdemote 128
Between these two commands, on the backup firewall, I see traffic
coming from WAN and DMZ, but almost nothing from LAN, so it may be
related to the LAN switch. I cannot see what the problem is though...

Here is the setup :

On both firewalls :
 - em0 is connected to WAN
 - em1 is connected to LAN
 - em2 is connected to DMZ
 - em3 is interconnected with a crossover cable, used for pfsync and
 rdist

WAN and DMZ connections are on the same switch, but on different
untagged VLANs (Procurve 2524) LAN is on a separate layer 3 switch
(Procurve 5300xl)

Another strange behavior :
With tcpdump, on the backup, I can see this traffic :
 - on em1 and em2, I see only carp advertisements to the configured
 unicast IP address and physical MAC address
 - on em3, I see only pfsync packets
 - but on em0, I see carp advertisements, but also a lot of traffic
 from the ISP router's MAC, to the virtual MAC (00:00:5e:00:01:01)
Which situation is normal? (em0 with lots of packets, or em1/em2 with
only carp advertisements) The only difference I see :
 - on em0, both firewalls and the ISP router are connected to the
 switch
 - on em1, both firewalls are connected to the L3 switch, which is
 also the router
 - on em2, there is no router, the firewalls communicate directly with
 hosts connected on the switch


Common configuration (public addresses anonymized, but the network
sizes are correct) :

/etc/mygate
192.0.2.1

/etc/sysctl.conf
net.inet.carp.preempt=1
net.inet.ip.forwarding=1

/etc/pf.conf (excerpt only)
ext_if  = em0
ext_if_carp = carp0
int_if  = em1
int_if_carp = carp1
dmz_if  = em2
dmz_if_carp = carp2
sync_if = em3
set skip on lo
set skip on $sync_if
pass quick on { $int_if, $ext_if, $dmz_if } inet proto carp keep state
(no-sync)


Firewall A (expected to be always master) :
OpenBSD 5.5 (GENERIC.MP) #315: Wed Mar  5 09:37:46 MST 2014
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP

/etc/hostname.em0
inet 192.168.3.9/30

/etc/hostname.em1
inet 192.168.3.1/29
!route add 192.168.0.0/16 192.168.3.5
!route add 172.16.0.0/12 192.168.3.5

/etc/hostname.em2
inet 192.168.3.13/30

/etc/hostname.em3
inet 192.168.3.17 255.255.255.252

/etc/hostname.carp0
advskew 0 carpdev em0 carppeer 192.168.3.10 pass secret1 state master
vhid 1 inet 192.0.2.2/28
alias 192.0.2.3/32
alias 192.0.2.4/32
alias 192.0.2.5/32

/etc/hostname.carp1
advskew 0 carpdev em1 carppeer 192.168.3.4 pass secret2 state master
vhid 2 inet 192.168.3.6/32

/etc/hostname.carp2
advskew 0 carpdev em2 carppeer 192.168.3.14 pass secret3 state master
vhid 3 inet 192.0.2.17/28
alias 192.0.2.29/32

/etc/hostname.pfsync0
up
syncdev em3
syncpeer 192.168.3.18


Firewall B (expected to be always backup) :
OpenBSD 5.6 (GENERIC.MP) #5: Thu Dec 11 09:51:08 CET 2014

 r...@stable-56-amd64.mtier.org:/binpatchng/work-binpatch56-amd64/src/sys/arch/amd64/compile/GENERIC.MP

/etc/hostname.em0
inet 192.168.3.10/30

/etc/hostname.em1
inet 192.168.3.4/29
!route add 192.168.0.0/16 192.168.3.5
!route add 172.16.0.0/12 192.168.3.5

/etc/hostname.em2
inet 192.168.3.14/30

/etc/hostname.em3
inet 192.168.3.18/30

/etc/hostname.carp0
advskew 200 carpdev em0 carppeer 192.168.3.9 pass secret1 state backup
vhid 1 inet 192.0.2.2/28
alias 192.0.2.3/32
alias 192.0.2.4/32
alias 192.0.2.5/32

/etc/hostname.carp1
advskew 200 carpdev em1 carppeer 192.168.3.1 pass secret2 state backup
vhid 2 inet 192.168.3.6/32

/etc/hostname.carp2
advskew 200 carpdev em2 carppeer 192.168.3.13 pass secret3 state
backup vhid 3 inet 192.0.2.17/28
alias 192.0.2.29/32

/etc/hostname.pfsync0
up
syncdev em3
syncpeer 192.168.3.17


This message is already long, but if any other information would be
helpful, I would be glad to provide it. Any help or suggestion is
appreciated. Thank you!

Sebastien


Sebastien,

Well, it's been many years since I ran carp, so I cannot actually help
with the carp config, but I can absolutely say that I have experienced a
lot of unexplainable weirdness with ProCurve switches, so I can
appreciate your suspicions there. I'll never buy another.

Re: carp hooks?

[etienne]

On 2015-01-05 19:51, Ted Unangst wrote:


I would like to know if there is any trigger in CARP, any way to run a
script on a CARP interface status change? I could monitor
/var/log/messages for that, but is there any cleaner, more efficient
way?


ifstated?


Thanks! Sorry, I promise, none of my searches helped me suspect even its 
existence.


Cheers,

--
Étienne

Re: carp hooks?

[Josh Grosse]

On 2015-01-05 18:38, etie...@magickarpet.org wrote:

On 2015-01-05 19:51, Ted Unangst wrote:

I would like to know if there is any trigger in CARP, any way to run 
a

script on a CARP interface status change? I could monitor
/var/log/messages for that, but is there any cleaner, more efficient
way?


ifstated?


Thanks! Sorry, I promise, none of my searches helped me suspect even
its existence.


It's mentioned three times in carp(4).  ;)

Re: carp hooks?

[Ted Unangst]

On Mon, Jan 05, 2015 at 19:43, etie...@magickarpet.org wrote:
 Hello list,
 
 I would like to know if there is any trigger in CARP, any way to run a
 script on a CARP interface status change? I could monitor
 /var/log/messages for that, but is there any cleaner, more efficient
 way?

ifstated?

Re: carp not reverting to master

[Marko Cupać]

On Thu, 02 Oct 2014 18:02:23 +0100
Andy a...@brandwatch.com wrote:

 Hi
 
 Try setting the advskew to a number greater than 200 and less then
 254. This seems to be the most stable.
 
 For best practice our primary runs with carp and pfsync values of
 '1'. And the backup runs with carp and pfsync values of '2'.
 
 We do this for two reasons.
 
 1) it is extremely stable!
 
 2) We found that CARP master is almost random/unstable when both 
 firewalls have the same value (esp '0'), because;
 
 When advbase is set to 0 the skew value alone is used to calculate
 how often advertisements are sent (the advertisement window) using
 this formula: Window in microseconds = advskew * 100 / 256
 
 E.g. 100 * 100 / 256 = 390625us
 
 So it would take much to cause a flip..
 
 Setting advbase to 1 on both is better as this is more stable if you 
 want to have the same carp demote counters..
 
 Good luck :)
 Andy

Andy,

thank you for the tip for increasing advskew value, I'm gonna try it out.

I had failover on another pair of firewalls, this time external ones,
running bgp. Carp is not reverting to master some 5 hours so far.

On master, while down, carp is demoted, pfsync is not:
 pacija@bgp1:~ $ ifconfig -g
 carp carp: carp demote count 1
 pacija@bgp1:~ $ ifconfig -g pfsync
 pfsync: carp demote count 0

On backup, while master, neither is demoted:
 pacija@bgp2:~ $ ifconfig -g
 carp carp: carp demote count 0
 pacija@bgp2:~ $ ifconfig -g pfsync
 pfsync: carp demote count 0

In /var/log/messages on downed master, I can see there was some
turbulence:
 Oct 14 15:21:19 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:21:19 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:21:22 bgp1 /bsd: carp1: state transition: BACKUP - MASTER
 Oct 14 15:21:22 bgp1 /bsd: carp2: state transition: BACKUP - MASTER
 Oct 14 15:22:52 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:22:52 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:22:53 bgp1 /bsd: carp3: state transition: MASTER - BACKUP
 Oct 14 15:23:02 bgp1 /bsd: carp3: state transition: BACKUP - MASTER
 Oct 14 15:23:03 bgp1 /bsd: carp1: state transition: BACKUP - MASTER
 Oct 14 15:23:03 bgp1 /bsd: carp2: state transition: BACKUP - MASTER
 Oct 14 15:23:41 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:23:41 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:23:41 bgp1 /bsd: carp3: state transition: MASTER - BACKUP
 Oct 14 15:23:54 bgp1 /bsd: carp3: state transition: BACKUP - MASTER
 Oct 14 15:23:56 bgp1 /bsd: carp2: state transition: BACKUP - MASTER
 Oct 14 15:23:56 bgp1 /bsd: carp1: state transition: BACKUP - MASTER
 Oct 14 15:26:04 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:26:04 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:26:04 bgp1 /bsd: carp3: state transition: MASTER - BACKUP

And in /var/log/daemon there is also bgp flapping at that time:
 Oct 14 15:22:53 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: directly 
 connected
 Oct 14 15:23:02 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: via 
 82.117.192.124
 Oct 14 15:23:41 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: directly 
 connected
 Oct 14 15:23:54 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: via 
 82.117.192.124
 Oct 14 15:26:04 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: directly 
 connected

82.117.192.124 is address of one of three carp interfaces.

I have 'demote carp' in bgpd.conf, so that master does not reclaim its
master role before bgp routes are up. The question remains, why is it
not reverting back to master once everything is ok?

-- 
Marko Cupać
https://www.mimar.rs

Re: carp not reverting to master

[Alan McKay]

On Wed, Oct 15, 2014 at 2:13 PM, Marko Cupać marko.cu...@mimar.rs wrote:
 Oct 14 15:21:19 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:21:19 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:21:22 bgp1 /bsd: carp1: state transition: BACKUP - MASTER
 Oct 14 15:21:22 bgp1 /bsd: carp2: state transition: BACKUP - MASTER
 Oct 14 15:22:52 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:22:52 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:22:53 bgp1 /bsd: carp3: state transition: MASTER - BACKUP
 Oct 14 15:23:02 bgp1 /bsd: carp3: state transition: BACKUP - MASTER
 Oct 14 15:23:03 bgp1 /bsd: carp1: state transition: BACKUP - MASTER
 Oct 14 15:23:03 bgp1 /bsd: carp2: state transition: BACKUP - MASTER

This looks to me like you have flapping taking place because of your
ifstated configuration.

Something is wrong with /etc/ifstated.conf on one end or the other.

-- 
“Don't eat anything you've ever seen advertised on TV”
 - Michael Pollan, author of In Defense of Food

Re: carp not reverting to master

[Andy Lemin]

Please excuse typos, sent from my phone

 On 15 Oct 2014, at 19:13, Marko Cupać marko.cu...@mimar.rs wrote:
 
 On Thu, 02 Oct 2014 18:02:23 +0100
 Andy a...@brandwatch.com wrote:
 
 Hi
 
 Try setting the advskew to a number greater than 200 and less then
 254. This seems to be the most stable.
 
 For best practice our primary runs with carp and pfsync values of
 '1'. And the backup runs with carp and pfsync values of '2'.
 
 We do this for two reasons.
 
 1) it is extremely stable!
 
 2) We found that CARP master is almost random/unstable when both 
 firewalls have the same value (esp '0'), because;
 
 When advbase is set to 0 the skew value alone is used to calculate
 how often advertisements are sent (the advertisement window) using
 this formula: Window in microseconds = advskew * 100 / 256
 
 E.g. 100 * 100 / 256 = 390625us
 
 So it would take much to cause a flip..
 
 Setting advbase to 1 on both is better as this is more stable if you 
 want to have the same carp demote counters..
 
 Good luck :)
 Andy
 
 Andy,
 
 thank you for the tip for increasing advskew value, I'm gonna try it out.
 
 I had failover on another pair of firewalls, this time external ones,
 running bgp. Carp is not reverting to master some 5 hours so far.
 
 On master, while down, carp is demoted, pfsync is not:
 pacija@bgp1:~ $ ifconfig -g
 carp carp: carp demote count 1
 pacija@bgp1:~ $ ifconfig -g pfsync
 pfsync: carp demote count 0
 
 On backup, while master, neither is demoted:
 pacija@bgp2:~ $ ifconfig -g
 carp carp: carp demote count 0
 pacija@bgp2:~ $ ifconfig -g pfsync
 pfsync: carp demote count 0
 
Hi, maybe in not reading your problem correctly but for as long as bgp1 has a 
demotion counter higher than bgp2 it will never go master.

 In /var/log/messages on downed master, I can see there was some
 turbulence:
 Oct 14 15:21:19 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:21:19 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:21:22 bgp1 /bsd: carp1: state transition: BACKUP - MASTER
 Oct 14 15:21:22 bgp1 /bsd: carp2: state transition: BACKUP - MASTER
 Oct 14 15:22:52 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:22:52 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:22:53 bgp1 /bsd: carp3: state transition: MASTER - BACKUP
 Oct 14 15:23:02 bgp1 /bsd: carp3: state transition: BACKUP - MASTER
 Oct 14 15:23:03 bgp1 /bsd: carp1: state transition: BACKUP - MASTER
 Oct 14 15:23:03 bgp1 /bsd: carp2: state transition: BACKUP - MASTER
 Oct 14 15:23:41 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:23:41 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:23:41 bgp1 /bsd: carp3: state transition: MASTER - BACKUP
 Oct 14 15:23:54 bgp1 /bsd: carp3: state transition: BACKUP - MASTER
 Oct 14 15:23:56 bgp1 /bsd: carp2: state transition: BACKUP - MASTER
 Oct 14 15:23:56 bgp1 /bsd: carp1: state transition: BACKUP - MASTER
 Oct 14 15:26:04 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:26:04 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:26:04 bgp1 /bsd: carp3: state transition: MASTER - BACKUP
 
 And in /var/log/daemon there is also bgp flapping at that time:
 Oct 14 15:22:53 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: directly 
 connected
 Oct 14 15:23:02 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: via 
 82.117.192.124
 Oct 14 15:23:41 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: directly 
 connected
 Oct 14 15:23:54 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: via 
 82.117.192.124
 Oct 14 15:26:04 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: directly 
 connected
 
 82.117.192.124 is address of one of three carp interfaces.
 
 I have 'demote carp' in bgpd.conf, so that master does not reclaim its
 master role before bgp routes are up. The question remains, why is it
 not reverting back to master once everything is ok?
 
 -- 
 Marko Cupać
 https://www.mimar.rs

Re: carp not reverting to master

[Andy Lemin]

Please excuse typos, sent from my phone

 On 15 Oct 2014, at 19:13, Marko Cupać marko.cu...@mimar.rs wrote:
 
 On Thu, 02 Oct 2014 18:02:23 +0100
 Andy a...@brandwatch.com wrote:
 
 Hi
 
 Try setting the advskew to a number greater than 200 and less then
 254. This seems to be the most stable.
 
 For best practice our primary runs with carp and pfsync values of
 '1'. And the backup runs with carp and pfsync values of '2'.
 
 We do this for two reasons.
 
 1) it is extremely stable!
 
 2) We found that CARP master is almost random/unstable when both 
 firewalls have the same value (esp '0'), because;
 
 When advbase is set to 0 the skew value alone is used to calculate
 how often advertisements are sent (the advertisement window) using
 this formula: Window in microseconds = advskew * 100 / 256
 
 E.g. 100 * 100 / 256 = 390625us
 
 So it would take much to cause a flip..
 
 Setting advbase to 1 on both is better as this is more stable if you 
 want to have the same carp demote counters..
 
 Good luck :)
 Andy
 
 Andy,
 
 thank you for the tip for increasing advskew value, I'm gonna try it out.
 
 I had failover on another pair of firewalls, this time external ones,
 running bgp. Carp is not reverting to master some 5 hours so far.
 
 On master, while down, carp is demoted, pfsync is not:
 pacija@bgp1:~ $ ifconfig -g
 carp carp: carp demote count 1
 pacija@bgp1:~ $ ifconfig -g pfsync
 pfsync: carp demote count 0
 
 On backup, while master, neither is demoted:
 pacija@bgp2:~ $ ifconfig -g
 carp carp: carp demote count 0
 pacija@bgp2:~ $ ifconfig -g pfsync
 pfsync: carp demote count 0
 
 In /var/log/messages on downed master, I can see there was some
 turbulence:
 Oct 14 15:21:19 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:21:19 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:21:22 bgp1 /bsd: carp1: state transition: BACKUP - MASTER
 Oct 14 15:21:22 bgp1 /bsd: carp2: state transition: BACKUP - MASTER
 Oct 14 15:22:52 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:22:52 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:22:53 bgp1 /bsd: carp3: state transition: MASTER - BACKUP
 Oct 14 15:23:02 bgp1 /bsd: carp3: state transition: BACKUP - MASTER
 Oct 14 15:23:03 bgp1 /bsd: carp1: state transition: BACKUP - MASTER
 Oct 14 15:23:03 bgp1 /bsd: carp2: state transition: BACKUP - MASTER
 Oct 14 15:23:41 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:23:41 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:23:41 bgp1 /bsd: carp3: state transition: MASTER - BACKUP
 Oct 14 15:23:54 bgp1 /bsd: carp3: state transition: BACKUP - MASTER
 Oct 14 15:23:56 bgp1 /bsd: carp2: state transition: BACKUP - MASTER
 Oct 14 15:23:56 bgp1 /bsd: carp1: state transition: BACKUP - MASTER
 Oct 14 15:26:04 bgp1 /bsd: carp2: state transition: MASTER - BACKUP
 Oct 14 15:26:04 bgp1 /bsd: carp1: state transition: MASTER - BACKUP
 Oct 14 15:26:04 bgp1 /bsd: carp3: state transition: MASTER - BACKUP
 
 And in /var/log/daemon there is also bgp flapping at that time:
 Oct 14 15:22:53 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: directly 
 connected
 Oct 14 15:23:02 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: via 
 82.117.192.124
 Oct 14 15:23:41 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: directly 
 connected
 Oct 14 15:23:54 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: via 
 82.117.192.124
 Oct 14 15:26:04 bgp1 bgpd[1380]: nexthop 82.117.192.124 now valid: directly 
 connected

Hi, You'll see these BGP messages as a result of the netstat -rn routing table 
changes when a box goes from master to backup or visa versa.

When a box is the backup, access to the carp IP will be in state connected as 
the routing table with have a MAC address for the CARP IP on the physical 
connected interface (taking you to the master), but when the box is the master 
there will be no MAC for the IP as its a local IP, hence the via.

I've always thought this problematic as this also causes issues with the BGP 
nexthop validation logic as when it's the master it considers the CARP IP not 
in the same broadcast domain as the subnet with the BGP peer. On old versions 
anyway, things may have changed..

 
 82.117.192.124 is address of one of three carp interfaces.
 
 I have 'demote carp' in bgpd.conf, so that master does not reclaim its
 master role before bgp routes are up. The question remains, why is it
 not reverting back to master once everything is ok?
 
 -- 
 Marko Cupać
 https://www.mimar.rs

Re: carp not reverting to master

[Andy]

nat1 will only preempt the nat2 after a fail-over to nat2 if the carp 
group and the pfsync group have the same demotion counter.

ifconfig -g carp
ifconfig -g pfsync

So if the failover which is happening for some unknown reason is 
affecting the demotion counters in anyway, preemption back to nat1 will 
not happen until you normalise the carp and pfsync group's demotion 
counters as you say..


Cheers, Andy.


On 02/10/14 10:24, Marko Cupać wrote:

Hi,

I have carp setup on two pairs of interfaces on our internal firewalls
that sit between private network and DMZ. The problem is that, for some
unknown reason, from time to time, carp fails over to nat2 (backup), and
does not revert to nat1 (master), until I manually carpdemote nat2.

If I understand carp well, my configuration should revert to master as
soon as it becomes available. If not, how can I achieve it?

Here's carp interfaces config:

nat1 (master):

pacija@nat1:~ $ sudo cat /etc/hostname.carp1
inet 192.168.225.6 255.255.255.248 192.168.225.7 \
vhid 1 pass mypass carpdev bnx0

pacija@nat1:~ $ sudo cat /etc/hostname.carp2
inet 193.53.106.32 255.255.255.0 193.53.106.255 \
vhid 2 pass mypass carpdev bnx1
inet alias 193.53.106.33 255.255.255.255
inet alias 193.53.106.34 255.255.255.255
inet alias 193.53.106.35 255.255.255.255
inet alias 193.53.106.36 255.255.255.255
inet alias 193.53.106.37 255.255.255.255
inet alias 193.53.106.38 255.255.255.255
inet alias 193.53.106.39 255.255.255.255

nat2 (backup):

pacija@nat2:~ $ sudo cat /etc/hostname.carp1
inet 192.168.225.6 255.255.255.248 192.168.225.7 \
vhid 1 advskew 100 pass mypass carpdev bnx0

pacija@nat2:~ $ sudo cat /etc/hostname.carp2
inet 193.53.106.32 255.255.255.0 193.53.106.255 \
vhid 2 advskew 100 pass mypass carpdev bnx1
inet alias 193.53.106.33 255.255.255.255
inet alias 193.53.106.34 255.255.255.255
inet alias 193.53.106.35 255.255.255.255
inet alias 193.53.106.36 255.255.255.255
inet alias 193.53.106.37 255.255.255.255
inet alias 193.53.106.38 255.255.255.255
inet alias 193.53.106.39 255.255.255.255

Re: carp not reverting to master

[Marko Cupać]

On Thu, 02 Oct 2014 10:37:19 +0100
Andy a...@brandwatch.com wrote:

 nat1 will only preempt the nat2 after a fail-over to nat2 if the
 carp group and the pfsync group have the same demotion counter.
 ifconfig -g carp
 ifconfig -g pfsync
 
 So if the failover which is happening for some unknown reason is 
 affecting the demotion counters in anyway, preemption back to nat1
 will not happen until you normalise the carp and pfsync group's
 demotion counters as you say..
 
 Cheers, Andy.

Hi Andy,

thank you for looking into it. At the moment nat1 is master, nat2 is
backup (desired situation).

On both firewalls demote carp for both groups (carp and pfsync) is 0:

pacija@nat1:~ $ ifconfig -g carp
carp: carp demote count 0
pacija@nat1:~ $ ifconfig -g pfsync
pfsync: carp demote count 0

pacija@nat2:~ $ ifconfig -g carp
carp: carp demote count 0
pacija@nat2:~ $ ifconfig -g pfsync
pfsync: carp demote count 0

If I reboot nat1, nat2 becomes master until nat1 reboots. After that, it
correctly hands master role to nat1. But in some situations (I don't
know what triggers them, hence to me their reason is unknown to me -
both firewalls are in same rack, switch, UPS etc.), nat1 hands master
role to nat2 and waits for days in backup role. I didn't look the value
of demote count for pfsync in this situation, but as for carp, they are
1 on nat1 (preferred master), and 0 on nat2 (preferred backup).

Is carp increasing demote counter on preferred master for some reason?
How can I make them normalize automatically?

Regards,
-- 
Marko Cupać
https://www.mimar.rs/

Re: carp not reverting to master

[Alan McKay]

You have not yet shown the output of ifconfig

Check the advskew values on the interfaces.

When carpdemote values are equal then advskew determines who is MASTER

Re: carp not reverting to master

[Marko Cupać]

On Thu, 2 Oct 2014 09:59:10 -0400
Alan McKay alan.mc...@gmail.com wrote:

 You have not yet shown the output of ifconfig
 
 Check the advskew values on the interfaces.
 
 When carpdemote values are equal then advskew determines who is MASTER
 

Hi Alan,

I have posted advskew values in initial mail (0 on masters, 100 on
backups).

What could be bumping carpdemote on master to 1?
-- 
Marko Cupać
https://www.mimar.rs/

Re: carp not reverting to master

[Alan McKay]

On Thu, Oct 2, 2014 at 11:03 AM, Marko Cupać marko.cu...@mimar.rs wrote:
 I have posted advskew values in initial mail (0 on masters, 100 on
 backups).

That shows me what they are supposed to be.

That does not show me what they actually are.

ifconfig output will show what they actually are.


-- 
“Don't eat anything you've ever seen advertised on TV”
 - Michael Pollan, author of In Defense of Food

Re: carp not reverting to master

[Andy]

Hi

Try setting the advskew to a number greater than 200 and less then 254. 
This seems to be the most stable.


For best practice our primary runs with carp and pfsync values of '1'. 
And the backup runs with carp and pfsync values of '2'.


We do this for two reasons.

1) it is extremely stable!

2) We found that CARP master is almost random/unstable when both 
firewalls have the same value (esp '0'), because;


When advbase is set to 0 the skew value alone is used to calculate how 
often advertisements are sent (the advertisement window) using this formula:

Window in microseconds = advskew * 100 / 256

E.g. 100 * 100 / 256 = 390625us

So it would take much to cause a flip..

Setting advbase to 1 on both is better as this is more stable if you 
want to have the same carp demote counters..


Good luck :)
Andy


On 02/10/14 16:08, Alan McKay wrote:

On Thu, Oct 2, 2014 at 11:03 AM, Marko Cupać marko.cu...@mimar.rs wrote:

I have posted advskew values in initial mail (0 on masters, 100 on
backups).

That shows me what they are supposed to be.

That does not show me what they actually are.

ifconfig output will show what they actually are.

Re: carp not reverting to master

[Andy]

PS; I would recommend setting the carpdemote to be a maximum (lowest) of 
1, becuase then if something happens to the primary box, and you can't 
get into it for some reason, at least you could set the carp demotion 
counters on the backup to 0' and remotely preempt your primary.




On 02/10/14 18:02, Andy wrote:

Hi

Try setting the advskew to a number greater than 200 and less then 
254. This seems to be the most stable.


For best practice our primary runs with carp and pfsync values of '1'. 
And the backup runs with carp and pfsync values of '2'.


We do this for two reasons.

1) it is extremely stable!

2) We found that CARP master is almost random/unstable when both 
firewalls have the same value (esp '0'), because;


When advbase is set to 0 the skew value alone is used to calculate 
how often advertisements are sent (the advertisement window) using 
this formula:

Window in microseconds = advskew * 100 / 256

E.g. 100 * 100 / 256 = 390625us

So it would take much to cause a flip..

Setting advbase to 1 on both is better as this is more stable if you 
want to have the same carp demote counters..


Good luck :)
Andy


On 02/10/14 16:08, Alan McKay wrote:
On Thu, Oct 2, 2014 at 11:03 AM, Marko Cupać marko.cu...@mimar.rs 
wrote:

I have posted advskew values in initial mail (0 on masters, 100 on
backups).

That shows me what they are supposed to be.

That does not show me what they actually are.

ifconfig output will show what they actually are.

Re: CARP cluster: howto keep pf.conf in sync?

[Peus, Christoph]

Hi all,

thanks for all your input to my small question about how to keep the pf.conf
in sync!
I have to care for exactly one firewall cluster, so I would like to avoid
complex tools for this task. I will probably use rdist.
Have fun!

Regards
Christoph

Private Universit?t Witten/Herdecke gGmbH
Alfred-Herrhausen-Stra?e 50
D - 58448 Witten

Homepage: http://www.uni-wh.de
Twitter: http://twitter.com/UniWH
Facebook: http://www.facebook.com/UniWH

Gesch?ftsf?hrung: Prof. Dr. Martin Butzlaff (Pr?sident), Dipl. oec. Jan Peter
Nonnenkamp (Kanzler)

Sitz der Gesellschaft: Witten
Handelsregister des Amtsgerichts Bochum Nr. HRB 8671

Re: CARP cluster: howto keep pf.conf in sync?

[Claer]

On Sat, Aug 02 2014 at 09:01, Nick Holland wrote:
 On 08/01/14 08:12, Claer wrote:
  On Mon, Jul 28 2014 at 07:23, Nick Holland wrote:
 ...
   I'll leave you to develop the script.
 
  My design philosophy:
  1) No additional hw, other than the two firewalls.
  2) EITHER machine should be able to act as master.
  3) EITHER machine should be able to provide all the info to rebuild the
  failed machine.
  4) Change control is good, just not how managers usually like to
  implement it.
  5) uses no other packages (rsync to move pf.conf around?  I don't think
  that's needed)
  
  Could you share it please ?
 
 well, no, in large part because I left the employment of that employer
 rather suddenly, and it seems I didn't save a copy of THAT script,
 though I do have some notes that will help (my DNS version).  (and yes,
 it's legit -- it wasn't a software company, and I had an understanding
 with the people that hired me that I could use any of the stuff I wrote
 however I wished.  The person who escorted me out I'm sure would
 disagree, but he got escorted out shortly afterwards.  BTW: if you ever
 find yourself being escorted out of a job for doing what you are
 confident is right, a great line is to politely ask, would you like me
 to deactivate my accounts, as you don't have anyone else left here to
 do it?  That's when the yelling began).
 
 Here are some code snippits that might be useful.  Nothing magical here,
 but there are a few tidbits I had to work out, but be forewarned, I
 probably did it the hard way (I'm proud of the ssh diff between two
 boxes, but that probably means I made it way too difficult.  This script
 is completely untested, I'm sure it won't work as is, and you get to
 provide your own error handling.  I'd call what I did an administration
 script not a user application.
 I'm assuming you have sudo access, and are SSH'ing to the first firewall
 with -A (agent forwarding) and have key access on both systems.
 
 # start.  Note the lack of #!/bin/sh, I'm not calling this a 
 # complete script!
 
 TMPLOG=/tmp/~config.log
 
 # /backup was a file system on a second disk in each FW.
 CHGLOG=/backup/changelog/`date +%Y-%m-%d-%H%M%S`.diff
 
 # Figure out who I am and who my partner machine is.
 # Our name -- easy.
 HERE=`hostname -s`
 # Other machine's name.  Assumption: machine names are in the form
 # *1 and *2, so that swapping the 1 and 2 will indicate the other machine.
 # This is a non-trivial assumption...but it works for us - fwa-1 - fwa-2
 OTHER=`echo $HERE |tr 12 21`
 
 # Generate a temp file with the diff between the old and new
 # file.  Should probably be with mktemp, but as there is a lack
 # of locking to protect against multiple users, there are bigger
 # issues here.
 echo %% Change by ${LOGNAME}@${HERE} on `date`: $TMPLOG
 echo $TMPLOG
 echo $TMPLOG
 
 ssh $OTHER sudo cat /etc/pf.conf | sudo diff -u - /etc/pf.conf $TMPLOG
 
 # Toss a marker to indicate when the change file was first made.
 touch ${TMPLOG}.tag
 chmod 664 ${TMPLOG}.tag  # makes it easier for other admins to delete.
 
 # Call up editor
 vi -c :3 $TMPLOG
 
 # If the temp log file is not newer than the .tag file, it apparently wasn't
 # edited, which means the commit was aborted.  Bail.  Note: IIRC, there were
 # some rough edges here.
 if [ ! $TMPLOG -nt ${TMPLOG}.tag ]; then
echo
echo
echo ** Sync with $OTHER aborted!! **
echo  NOTE: DNS servers are likely out of sync!
echo
rm $TMPLOG ${TMPLOG}.tag
exit
 fi
 
 Save the change log HERE.
 mv $TMPLOG $CHGLOG
 
 # Copy stuff over to $OTHER server
 echo Syncing with other server
 scp $CHGLOG $OTHER:$CHGLOG
 scp /etc/pf.conf $OTHER:/tmp/pf.conf 
 ssh $OTHER sudo mv /tmp/pf.conf /etc
 
 # install. you DID test this, right?  Note the lack of error handling!
 ssh $OTHER sudo pfctl -f /etc/pf.conf
 
 rm ${TMPLOG}.tag
 
 
 That's pretty much the strategy.  Lots of site specific assumptions,
 lots of things that could be done better in the script.  As noted,
 one major flaw is the handling when two admins are making
 changes at the same time, but then, at this site, the two of us were
 both familiar with the OpenBSD ways, and always tried to get an ok
 from the other before making a change, which ensured that we both
 knew a change was coming.  Its handling of issues like admin A starts
 but never finishes the update, then B comes along and does an update
 are crude, but if you write your own, you know what the errors mean.
 
 If I were doing this again, I'd probably put in some kind of
 comparison of hostname.carp* files, as we found if those are not in
 sync ugly things happened.  
 
 My favorite part, though is the changes are almost self-documenting,
 so easy that the administrator won't object, and having the change
 diff stuffed in your face is just an overall good plan, I think.
 And, to find why a particular line was made, use grep to find
 when the line was changed/added, and look at the commit message.
 
 I've been told I 

Re: carp setup firewall

[Henning Brauer]

* Kim Zeitler kim.zeit...@konzept-is.de [2014-07-25 11:19]:
 we have a similar setup here, with only a /29 range of external addresses.
 Until now, we have had no problems so far running this using only one
 external carp IF (using a private IP) and adding all external addresses
 as aliases. But we do not use bi-nat for our DMZ Servers.

there really is nothing wrong with aliases on carp interfaces.

you ahve to keep them in sync of course. just like the vhid and the
passphrase...

-- 
Henning Brauer, h...@bsws.de, henn...@openbsd.org
BS Web Services GmbH, http://bsws.de, Full-Service ISP
Secure Hosting, Mail and DNS. Virtual  Dedicated Servers, Root to Fully Managed
Henning Brauer Consulting, http://henningbrauer.com/

Re: CARP cluster: howto keep pf.conf in sync?

[R0me0 ***]

I wrote a little script sometime ago and it run from crontab every 5 min
and do:

check and generate md5 of important files like hostname.if , pf include
files, etc ...

All necessaries modification is monitored natively by OpenBSD, but there is
an ossec in deployment as well.

ifstated is used to invert from/to ( always from master to slave )

I hope this help you =)



2014-07-28 8:50 GMT-03:00 Peus, Christoph christoph.p...@uni-wh.de:

 Hi all,



 is there a standard or recommended way to keep the pf.conf on the CARP
 cluster
 members in sync?

 Thanks!

 Regards
 Christoph

 --
 Christoph Peus
 Universität Witten/Herdecke
 Bereich Informationstechnologie
 Tel:  +49 2302 926-212
 Fax: +49 2302 926-44857
 mailto:christoph.p...@uni-wh.de











 Private Universität Witten/Herdecke gGmbH
 Alfred-Herrhausen-Straße 50
 D - 58448 Witten

 Homepage: http://www.uni-wh.de
 Twitter: http://twitter.com/UniWH
 Facebook: http://www.facebook.com/UniWH

 Geschäftsführung: Prof. Dr. Martin Butzlaff (Präsident), Dipl. oec. Jan
 Peter
 Nonnenkamp (Kanzler)

 Sitz der Gesellschaft: Witten
 Handelsregister des Amtsgerichts Bochum Nr. HRB 8671

Re: CARP cluster: howto keep pf.conf in sync?

[sven falempin]

On Fri, Aug 1, 2014 at 4:56 AM, R0me0 *** knight@gmail.com wrote:
 I wrote a little script sometime ago and it run from crontab every 5 min
 and do:

 check and generate md5 of important files like hostname.if , pf include
 files, etc ...


doh !
this is done in daily/security
look at /etc/changelist


 All necessaries modification is monitored natively by OpenBSD, but there is
 an ossec in deployment as well.

 ifstated is used to invert from/to ( always from master to slave )

 I hope this help you =)



 2014-07-28 8:50 GMT-03:00 Peus, Christoph christoph.p...@uni-wh.de:

 Hi all,



 is there a standard or recommended way to keep the pf.conf on the CARP
 cluster
 members in sync?

 Thanks!

 Regards
 Christoph

 --
 Christoph Peus
 Universität Witten/Herdecke
 Bereich Informationstechnologie
 Tel:  +49 2302 926-212
 Fax: +49 2302 926-44857
 mailto:christoph.p...@uni-wh.de











 Private Universität Witten/Herdecke gGmbH
 Alfred-Herrhausen-Straße 50
 D - 58448 Witten

 Homepage: http://www.uni-wh.de
 Twitter: http://twitter.com/UniWH
 Facebook: http://www.facebook.com/UniWH

 Geschäftsführung: Prof. Dr. Martin Butzlaff (Präsident), Dipl. oec. Jan
 Peter
 Nonnenkamp (Kanzler)

 Sitz der Gesellschaft: Witten
 Handelsregister des Amtsgerichts Bochum Nr. HRB 8671




-- 
-
() ascii ribbon campaign - against html e-mail
/\

Re: CARP cluster: howto keep pf.conf in sync?

[Claer]

Hello,

On Mon, Jul 28 2014 at 07:23, Nick Holland wrote:
 On 07/28/14 07:50, Peus, Christoph wrote:
  Hi all,
  
  
  
  is there a standard or recommended way to keep the pf.conf on the CARP 
  cluster
  members in sync?
  
  Thanks!
 
 No one standard or recommended way, but lots of ideas, as you can see.
 
 Here's mine, but for the moment, I'll leave you to develop the script.
 
 My design philosophy:
 1) No additional hw, other than the two firewalls.
 2) EITHER machine should be able to act as master.
 3) EITHER machine should be able to provide all the info to rebuild the
 failed machine.
 4) Change control is good, just not how managers usually like to
 implement it.
 5) uses no other packages (rsync to move pf.conf around?  I don't think
 that's needed)

Could you share it please ?


 So...  I wrote a relatively simple little script which
 * Figures out which the other machine is
 * does a diff -u of the changes between the local machine and the
 other machine (assuming the other machine is the old config)
 * Displays the diff to the user, and asks you to explain the change.
 * records the diff and your explanation to a file with a date and time
 stamp as a file name into a change log directory.
 * copies the pf.conf and the change log file to the corresponding
 directory in the other machine.
 * pfctl -f /etc/pf.conf's the other machine.
 
 So...you make a change on one box (EITHER!), test it, when satisified,
 you run the sync script.  It compares the changed file to the other
 system, shows you the diff, and you can:
 1) comment it and save it to both
 2) Realize you made a typo, and deleted something you didn't intend to
 or fat-fingered something you didn't intend to, fix.
 3) Realize that you made some other changes that weren't sync'd on
 either machine
 4) etc.
 
 The script is identical between machines, so if you lose EITHER
 firewall, the other can be used to rebuild the missing system, including
 the history.
 
 If something goes horribly wrong, you just dig out the history file, and
 revert the change.  If something goes horribly wrong before you sync it,
 log into the other firewall, and push the changes back.
 
 Wonder why a rule is in the firewall? Look back through the change log
 and read the comments.
 
 I've done the same thing with DNS zone files and config files, (in my
 opinion) better than the BIND master/slave model -- set up each node
 as a master, and sync the data through scripts like this.
 
 Nick.

Claer

Re: CARP cluster: howto keep pf.conf in sync?

[Giancarlo Razzolini]

On 01-08-2014 09:07, sven falempin wrote:
 doh !
 this is done in daily/security
 look at /etc/changelist
It's not md5, it's sha256. md5 should not be used anymore. But what
Romeo does is to run a script from cron every 5 minutes. Daily runs,
obviously, daily. It's not suited for the task at hand. But if you ask
me, I don't like this reactive approach. I use git repo with hooks to
apply changes as they are pushed to the central repository. But that's
the nicest about *unix. There are lots of ways of doing things. You can
copy things manually, create scripts to semi-automate things, use
version control, use puppet and friends, etc. It's all about what you
are most comfortably with.

Cheers,

--
Giancarlo Razzolini
GPG: 4096R/77B981BC

[demime 1.01d removed an attachment of type application/pkcs7-signature which 
had a name of smime.p7s]

Re: CARP cluster: howto keep pf.conf in sync?

[sven falempin]

On Fri, Aug 1, 2014 at 8:22 AM, Giancarlo Razzolini
grazzol...@gmail.com wrote:
 On 01-08-2014 09:07, sven falempin wrote:
 doh !
 this is done in daily/security
 look at /etc/changelist
 It's not md5, it's sha256. md5 should not be used anymore. But what
 Romeo does is to run a script from cron every 5 minutes. Daily runs,
 obviously, daily. It's not suited for the task at hand. But if you ask
 me, I don't like this reactive approach. I use git repo with hooks to
 apply changes as they are pushed to the central repository. But that's
 the nicest about *unix. There are lots of ways of doing things. You can
 copy things manually, create scripts to semi-automate things, use
 version control, use puppet and friends, etc. It's all about what you
 are most comfortably with.

actually if you dont put a + it is plain diff and a backup in /var,
the security could be run more often (it is called in the cron), and
because the script is present there is no need to write it again.

I asked a long time ago if someone know a versionning system that does not
rely on a local copy like .git .svn etc...

i found some exotic weard stuff but nothing fancy.



 Cheers,

 --
 Giancarlo Razzolini
 GPG: 4096R/77B981BC





-- 
-
() ascii ribbon campaign - against html e-mail
/\

Re: CARP cluster: howto keep pf.conf in sync?

[Giancarlo Razzolini]

On 01-08-2014 09:32, sven falempin wrote:
 actually if you dont put a + it is plain diff and a backup in /var,
 the security could be run more often (it is called in the cron), and
 because the script is present there is no need to write it again.
security(8) is called by daily(8). You could call it from other scripts.
But I advise against changing daily(8) to run more often.

 I asked a long time ago if someone know a versionning system that does
not
 rely on a local copy like .git .svn etc...

 i found some exotic weard stuff but nothing fancy.
What do you mean rely on a local copy? I don't make symlinks to files in
my clonned repo. I instead have a script that copies the files to /etc
or whatever is their place. The most amazing thing of git is just the
fact that your repo is a full copy. If something should happen to your
central repo, your backups, and your carp master node(doomsday?), you
can recover everything from a single cloned repo.

Cheers,

--
Giancarlo Razzolini
GPG: 4096R/77B981BC

[demime 1.01d removed an attachment of type application/pkcs7-signature which 
had a name of smime.p7s]

Re: CARP cluster: howto keep pf.conf in sync?

[Zach Leslie]

  Configuration management tools, like Puppet, can quickly abstract
  knowledge of a particular technology away from the user and isolate
  understanding for said technology to a smaller group of people with
  those skills.  This is the nature of technology, though, is it not?
  Abstractions built on abstractions, packages including libraries, etc.
  There is an inherent trust in the tools and, more importantly, the
  authors of those tools.  This does not mean that the recipes (as you
  put it) are inherently bad, or manage a system poorly, or that great
  care cannot be taken to manage a system effectively, and securely.  Ha,
  but there is also lots of bad code in the world.  Such is life.
 Of course. But the problem is a false sense of rightness and security
 that these tools give to people that are not aware of all the
 implications. If you read a recipe and does not understand all that it
 does, then how can you be sure it won't mess with your system.

I agree, though, I'd extend that sentiment far beyond config management.

  The trust in a system's authors is one of the major reasons I use
  OpenBSD in critical infrastructure without having to know anything about
  how the compiler functions at its core.  Without this trust, we'd still
  be smacking coconuts against rocks instead of building bridges to the
  UberTech, so to speak.
 Don't get me wrong. I like these tools. But, for a few servers, I prefer
 to manage them directly. I'm warning that these tools need proper use,
 they are not a one size fits all solution.

No doubt.  There is also something to be said for beautifully
handcrafted config files.


--
Zach

[demime 1.01d removed an attachment of type application/pgp-signature]

Re: CARP cluster: howto keep pf.conf in sync?

[R0me0 ***]

Hi Giancarlo,
I would like to thank your background (:
Yes the important files is included @changelist and it's sha256, but as
firewall rules has modifications during all time, another nodes need be
updated. So, it's because of this I run the script every 5 min and I sync
it using SCP.

* My script runs independent of daily scripts * and the hash is md5.

Thank you @misc .















2014-08-01 9:22 GMT-03:00 Giancarlo Razzolini grazzol...@gmail.com:

 On 01-08-2014 09:07, sven falempin wrote:
  doh !
  this is done in daily/security
  look at /etc/changelist
 It's not md5, it's sha256. md5 should not be used anymore. But what
 Romeo does is to run a script from cron every 5 minutes. Daily runs,
 obviously, daily. It's not suited for the task at hand. But if you ask
 me, I don't like this reactive approach. I use git repo with hooks to
 apply changes as they are pushed to the central repository. But that's
 the nicest about *unix. There are lots of ways of doing things. You can
 copy things manually, create scripts to semi-automate things, use
 version control, use puppet and friends, etc. It's all about what you
 are most comfortably with.

 Cheers,

 --
 Giancarlo Razzolini
 GPG: 4096R/77B981BC

Re: CARP cluster: howto keep pf.conf in sync?

[Nick Holland]

On 08/01/14 08:12, Claer wrote:
 On Mon, Jul 28 2014 at 07:23, Nick Holland wrote:
...
  I'll leave you to develop the script.

 My design philosophy:
 1) No additional hw, other than the two firewalls.
 2) EITHER machine should be able to act as master.
 3) EITHER machine should be able to provide all the info to rebuild the
 failed machine.
 4) Change control is good, just not how managers usually like to
 implement it.
 5) uses no other packages (rsync to move pf.conf around?  I don't think
 that's needed)
 
 Could you share it please ?

well, no, in large part because I left the employment of that employer
rather suddenly, and it seems I didn't save a copy of THAT script,
though I do have some notes that will help (my DNS version).  (and yes,
it's legit -- it wasn't a software company, and I had an understanding
with the people that hired me that I could use any of the stuff I wrote
however I wished.  The person who escorted me out I'm sure would
disagree, but he got escorted out shortly afterwards.  BTW: if you ever
find yourself being escorted out of a job for doing what you are
confident is right, a great line is to politely ask, would you like me
to deactivate my accounts, as you don't have anyone else left here to
do it?  That's when the yelling began).

Here are some code snippits that might be useful.  Nothing magical here,
but there are a few tidbits I had to work out, but be forewarned, I
probably did it the hard way (I'm proud of the ssh diff between two
boxes, but that probably means I made it way too difficult.  This script
is completely untested, I'm sure it won't work as is, and you get to
provide your own error handling.  I'd call what I did an administration
script not a user application.
I'm assuming you have sudo access, and are SSH'ing to the first firewall
with -A (agent forwarding) and have key access on both systems.

# start.  Note the lack of #!/bin/sh, I'm not calling this a 
# complete script!

TMPLOG=/tmp/~config.log

# /backup was a file system on a second disk in each FW.
CHGLOG=/backup/changelog/`date +%Y-%m-%d-%H%M%S`.diff

# Figure out who I am and who my partner machine is.
# Our name -- easy.
HERE=`hostname -s`
# Other machine's name.  Assumption: machine names are in the form
# *1 and *2, so that swapping the 1 and 2 will indicate the other machine.
# This is a non-trivial assumption...but it works for us - fwa-1 - fwa-2
OTHER=`echo $HERE |tr 12 21`

# Generate a temp file with the diff between the old and new
# file.  Should probably be with mktemp, but as there is a lack
# of locking to protect against multiple users, there are bigger
# issues here.
echo %% Change by ${LOGNAME}@${HERE} on `date`: $TMPLOG
echo $TMPLOG
echo $TMPLOG

ssh $OTHER sudo cat /etc/pf.conf | sudo diff -u - /etc/pf.conf $TMPLOG

# Toss a marker to indicate when the change file was first made.
touch ${TMPLOG}.tag
chmod 664 ${TMPLOG}.tag  # makes it easier for other admins to delete.

# Call up editor
vi -c :3 $TMPLOG

# If the temp log file is not newer than the .tag file, it apparently wasn't
# edited, which means the commit was aborted.  Bail.  Note: IIRC, there were
# some rough edges here.
if [ ! $TMPLOG -nt ${TMPLOG}.tag ]; then
   echo
   echo
   echo ** Sync with $OTHER aborted!! **
   echo  NOTE: DNS servers are likely out of sync!
   echo
   rm $TMPLOG ${TMPLOG}.tag
   exit
fi

Save the change log HERE.
mv $TMPLOG $CHGLOG

# Copy stuff over to $OTHER server
echo Syncing with other server
scp $CHGLOG $OTHER:$CHGLOG
scp /etc/pf.conf $OTHER:/tmp/pf.conf 
ssh $OTHER sudo mv /tmp/pf.conf /etc

# install. you DID test this, right?  Note the lack of error handling!
ssh $OTHER sudo pfctl -f /etc/pf.conf

rm ${TMPLOG}.tag


That's pretty much the strategy.  Lots of site specific assumptions,
lots of things that could be done better in the script.  As noted,
one major flaw is the handling when two admins are making
changes at the same time, but then, at this site, the two of us were
both familiar with the OpenBSD ways, and always tried to get an ok
from the other before making a change, which ensured that we both
knew a change was coming.  Its handling of issues like admin A starts
but never finishes the update, then B comes along and does an update
are crude, but if you write your own, you know what the errors mean.

If I were doing this again, I'd probably put in some kind of
comparison of hostname.carp* files, as we found if those are not in
sync ugly things happened.  

My favorite part, though is the changes are almost self-documenting,
so easy that the administrator won't object, and having the change
diff stuffed in your face is just an overall good plan, I think.
And, to find why a particular line was made, use grep to find
when the line was changed/added, and look at the commit message.

I've been told I should use rcs or cvs or similar...but I really
prefer the one change per file and all text file format.

Nick.

Re: CARP cluster: howto keep pf.conf in sync?

[Zach Leslie]

On Tue, Jul 29, 2014 at 02:41:36PM +0100, Andy wrote:
 Puppet is definatly a sledge hammer approach, but if you have lots of
 firewalls its great.

Not to mention, you can use it for your other non-firewall systems as
well.

 Another nice example of an appropriate application is that by using
 PuppetDB, a full IPSec VPN mesh is built automatically by puppet between
 every firewall according to the subnets behind each firewall pair. So if I
 add a single new subnet behind a remote office firewall, the 12 odd extra
 tunnels all get created automatically.

 But unless you are wanting to do stuff like that, then yes, I completely
 agree with Nick puppet is major over kill..

For even a small environment, being able to ERB template your PF configs
is really nice.  You can use a master if you want, or you can just do
standalone puppet apply, where you ship all of the code needed to each
system that needs it.

I'm a Puppet user for more than just firewall systems, which allows me
to take a given node, say another server, and insert its IP into a table
on the firewall, completely dynamicly without having to statically set
the IPs in pf.conf.  There are lots of interesting things you can do
with Puppet that allow you build dynamic tables based on the
classification of other systems in your environment.

For the curious: https://github.com/xaque208/puppet-bsd

I started working on this over the last year, with the idea in mind that
I'd eventually be able to define the high level components I care to manage on
a
given BSD system and things would just happen.  Eventually things like
OSPF, DHCP configs etc.  There are plenty of modules that work on Linux,
but not as many that work on BSD, OpenBSD even more so.

--
Zach

[demime 1.01d removed an attachment of type application/pgp-signature]