Re: CARP, BGP and OpenBSD routers - integration tips/suggestions needed

[Bob Jones]

Hi Bryan,

Thank you for the great message.  I will re-read it in more detail
over the next few days and have a go at getting all the pieces of the
jigsaw put together !

Thanks again.

On 11 December 2016 at 18:12, Bryan Vyhmeister  wrote:
> On Sun, Dec 11, 2016 at 09:45:08AM +, Bob Jones wrote:
>> I have a planned network topology that will run on OpenBSD that (at
>> the moment) will constitute of three boxes :
>>
>> 1 x Router (Openbsd running bgpd for connection to the outside world)
>> 2 x Firewalls (running Openbsd)
>>
>> I can't quite figure out the best way to deal with the "external" side
>> of the firewalls ? (Obviously the "internal" side would be CARP).
>
> The missing piece here is OSPF. The paper below is what I used as my
> template to setup my network which is very much like your design. I then
> added the CARP configuration which I will explain below.
>
> http://www.openbsd.org/papers/linuxtag06-network.pdf
>
>> At the moment, since the devices are located in the same rack, I am
>> thinking of running a patch cable directly from each firewall to two
>> ports on the Router (i.e. F1a -> R1a and F2a to R1b).  The reason for
>> this is to avoid going via a switch and adding a point of failure
>> (yes, I know, I only have one "router" but hopefully that will
>> change in the not too distant future !)
>
> I have an external router running BGP with my provider with four
> ethernet ports. The first interface (em0) is connected to my provider.
> The other ports (em1, em2, and em3) are all part of bridge0 which is
> what my other two routers are connected to. The internal IP address
> (which is .1 of a /29 and the beginning of my /24) of my external router
> exists on vether0 also added to bridge0. I run iBGP and OSPF between the
> three routers as in the paper above. That means the first internal
> router would have .3 on its em0 and the second internal router would
> have .4 on its em1. I am actually changing out to use a switch because
> once in a while the external router stops seeing OSPF from the internal
> routers. I can't be positive it has anything to do with bridge(4)
> because the routers have not been upgraded recently past 5.8 and there
> have been improvements to lots of areas. This week I am upgrading them
> to 6.0-stable and replacing some hardware. I have this same setup in a
> datacenter as well all running 6.0-stable that uses a switch and has
> worked perfectly for several years.
>
>> The problem is I can't quite figure out the OpenBSD software
>> configuration for that concept and how it inter-relate with CARP
>> running on the "internal" side of the firewalls ?  Should I be running
>> OSPF ? iBGP ?  Or something else (switchd ? vether ?)
>
> Like I mentioned above, you want BGP to your provider on the external
> router. On its internal interface use iBGP and OSPF to the other two
> routers. You can use vether(4) and bridge(4) on the external router's
> internal interfaces like I did which seems to work fairly well. On the
> two internal routers, the key point for CARP to work is to use the
> "demote carp" option as documented in ospfd.conf(5) and use the carp
> interface(s) rather than the physical in ospfd.conf on the internal side
> of the internal routers. You also have to have a link between the
> internal routers for pfsync(4) and an OSPF link. Here is a simplified
> snippet from my ospfd.conf on one of the internal routers.
>
> area 0.0.0.0 {
> demote carp
> interface em0 { metric 10 }
> interface em1 { metric 20 }
> interface carp2 { passive }
> }
>
> In this case, em0 connects to the external router. Interface em1 is a
> cable between the two internal routers which provides both a /30 link
> between them for OSPF and also pfsync for CARP to work correctly. I
> simplified my snippet above because I have some other things working
> that would complicate your setup. In a setup like you want, carp2 would
> correspond to em2 for example. I am using a /25 for the internal network
> using CARP. So carp2 has the .125 IP address shared bewteen both
> internal routers, em2 on the first internal router has .126, and em2 on
> the second internal router has .127 to allow CARP to work correctly.
>
> Hopefully this helps you get things going. This setup works very well
> for me at multiple sites and can easily be expanded by adding another
> external router to another provider in the future.
>
> Bryan

Re: CARP, BGP and OpenBSD routers - integration tips/suggestions needed

[Bryan Vyhmeister]

On Sun, Dec 11, 2016 at 09:45:08AM +, Bob Jones wrote:
> I have a planned network topology that will run on OpenBSD that (at
> the moment) will constitute of three boxes :
> 
> 1 x Router (Openbsd running bgpd for connection to the outside world)
> 2 x Firewalls (running Openbsd)
> 
> I can't quite figure out the best way to deal with the "external" side
> of the firewalls ? (Obviously the "internal" side would be CARP).

The missing piece here is OSPF. The paper below is what I used as my
template to setup my network which is very much like your design. I then
added the CARP configuration which I will explain below.

http://www.openbsd.org/papers/linuxtag06-network.pdf

> At the moment, since the devices are located in the same rack, I am
> thinking of running a patch cable directly from each firewall to two
> ports on the Router (i.e. F1a -> R1a and F2a to R1b).  The reason for
> this is to avoid going via a switch and adding a point of failure
> (yes, I know, I only have one "router" but hopefully that will
> change in the not too distant future !)

I have an external router running BGP with my provider with four
ethernet ports. The first interface (em0) is connected to my provider.
The other ports (em1, em2, and em3) are all part of bridge0 which is
what my other two routers are connected to. The internal IP address
(which is .1 of a /29 and the beginning of my /24) of my external router
exists on vether0 also added to bridge0. I run iBGP and OSPF between the
three routers as in the paper above. That means the first internal
router would have .3 on its em0 and the second internal router would
have .4 on its em1. I am actually changing out to use a switch because
once in a while the external router stops seeing OSPF from the internal
routers. I can't be positive it has anything to do with bridge(4)
because the routers have not been upgraded recently past 5.8 and there
have been improvements to lots of areas. This week I am upgrading them
to 6.0-stable and replacing some hardware. I have this same setup in a
datacenter as well all running 6.0-stable that uses a switch and has
worked perfectly for several years.

> The problem is I can't quite figure out the OpenBSD software
> configuration for that concept and how it inter-relate with CARP
> running on the "internal" side of the firewalls ?  Should I be running
> OSPF ? iBGP ?  Or something else (switchd ? vether ?)

Like I mentioned above, you want BGP to your provider on the external
router. On its internal interface use iBGP and OSPF to the other two
routers. You can use vether(4) and bridge(4) on the external router's
internal interfaces like I did which seems to work fairly well. On the
two internal routers, the key point for CARP to work is to use the
"demote carp" option as documented in ospfd.conf(5) and use the carp
interface(s) rather than the physical in ospfd.conf on the internal side
of the internal routers. You also have to have a link between the
internal routers for pfsync(4) and an OSPF link. Here is a simplified
snippet from my ospfd.conf on one of the internal routers.

area 0.0.0.0 {
demote carp
interface em0 { metric 10 }
interface em1 { metric 20 }
interface carp2 { passive }
}

In this case, em0 connects to the external router. Interface em1 is a
cable between the two internal routers which provides both a /30 link
between them for OSPF and also pfsync for CARP to work correctly. I
simplified my snippet above because I have some other things working
that would complicate your setup. In a setup like you want, carp2 would
correspond to em2 for example. I am using a /25 for the internal network
using CARP. So carp2 has the .125 IP address shared bewteen both
internal routers, em2 on the first internal router has .126, and em2 on
the second internal router has .127 to allow CARP to work correctly.

Hopefully this helps you get things going. This setup works very well
for me at multiple sites and can easily be expanded by adding another
external router to another provider in the future.

Bryan

CARP, BGP and OpenBSD routers - integration tips/suggestions needed

[Bob Jones]

Hi,

I have a planned network topology that will run on OpenBSD that (at
the moment) will constitute of three boxes :

1 x Router (Openbsd running bgpd for connection to the outside world)
2 x Firewalls (running Openbsd)

I can't quite figure out the best way to deal with the "external" side
of the firewalls ? (Obviously the "internal" side would be CARP).

At the moment, since the devices are located in the same rack, I am
thinking of running a patch cable directly from each firewall to two
ports on the Router (i.e. F1a -> R1a and F2a to R1b).  The reason for
this is to avoid going via a switch and adding a point of failure
(yes, I know, I only have one "router" but hopefully that will
change in the not too distant future !)

The problem is I can't quite figure out the OpenBSD software
configuration for that concept and how it inter-relate with CARP
running on the "internal" side of the firewalls ?

Should I be running OSPF ? iBGP ?  Or something else (switchd ? vether ?)

I'm terribly confused !

Bob