Re: isakmpd multiple tunnels

[Tim Pushor]

Thanks for the response.

I should have been more clear. I am using isakmpd.conf and want to 
support multiple tunnels. Am I able to just add additional tunnels/lines 
under the [Phase 1] block that points to another relevant ISPEC 
configuration?


Anyone?

Thanks,
Tim

Hans-Joerg Hoexer wrote:

On Thu, Apr 12, 2007 at 11:25:49AM -0600, Tim Pushor wrote:
  

Hi friends,

I'm looking to add another IPSEC connection to my openbsd 3.9 firewall. 
All examples I've seen are a single connection (phase 1). To support 
multiple vpn's tunnels, is it as simple as adding additional lines under 
[Phase 1] pointing to the new phase1 configuration block?



yes.  However, please take a look at ipsecctl(8) and ipsec.conf(5).

HJ.

isakmpd multiple tunnels

[Tim Pushor]

Hi friends,

I'm looking to add another IPSEC connection to my openbsd 3.9 firewall. 
All examples I've seen are a single connection (phase 1). To support 
multiple vpn's tunnels, is it as simple as adding additional lines under 
[Phase 1] pointing to the new phase1 configuration block?


Thanks!

Re: Problem with MTU IPSec VPN

[Tim Pushor]

Hi Darren,

Just want to say thank you. You helped solve a problem thats been 
hounding me for a while now. It was in fact the smartdefence, but it was 
the number of fragmented packets allowed in a certain timespan.


The problem was that I don't have access to the device and thus had to 
troubleshoot at one end of the connection until I was pretty sure it was 
the other side. Thats a crappy position to be in. Your help, coupled 
with the fact that I could see the pings arriving at the OpenBSD server 
and the response being sent out pointed to a problem on the other end.


Now I just have to figure out why Path MTU discovery isn't working, but 
thats minor at this point.


I'd love to send you a pizza of your choice. Please drop me an email and 
it'll be done. I'm serious. I'm SO relieved.


Thanks,
Tim


Darren Spruell wrote:

On 2/19/07, Tim Pushor [EMAIL PROTECTED] wrote:

Hi all,

I'm getting to the point where I don't really know where to turn. I am
having a weird problem with an OpenBSD server/firewall that has a
permament IPSec tunnel to a checkpoint embedded security device. The
problem is, that half the time large packets can't get through. I've
trial and error'ed (via windows ping -l) that ping packets 1306 bytes
get through all the time, while packets  1306 (even 1307) only get
through half the time. Not half the time like 50% loss, but like it
works for hours, then doesn't for 10 minutes.


If that Check Point device has SmartDefense enabled, it has rules that
futz with ICMP packets larger than some threshold. See if you have any
of that mojo going on.

DS

Problem with MTU IPSec VPN

[Tim Pushor]

Hi all,

I'm getting to the point where I don't really know where to turn. I am 
having a weird problem with an OpenBSD server/firewall that has a 
permament IPSec tunnel to a checkpoint embedded security device. The 
problem is, that half the time large packets can't get through. I've 
trial and error'ed (via windows ping -l) that ping packets 1306 bytes 
get through all the time, while packets  1306 (even 1307) only get 
through half the time. Not half the time like 50% loss, but like it 
works for hours, then doesn't for 10 minutes.


I only have control of one half of the connection (unfortunately), and 
am kind of lost. I have a rudimentary understanding of IP, so do 
understand things like MTU's  fragementation, and things like VPN 
adding packet overhead that reduces the effective MTU. I just don't 
really know where to start tracking this down. I guess I don't 
understand enough (any?) about how this part of the tunnel works  under 
the covers.


Can anyone help maybe point me in a direction? My ruleset is default 
deny with log, and nothing is being dropped. pfctl -x loud doesn't 
reveal anything. I pass everything to/from the VPN. My network 
configuration is unfortunately kind of complex. I have vlan's and carp's 
on those vlan's so I'll refrain from trying to describe the setup unless 
its necessary.


Hopefully thanks in advance :)
Tim

watch traffic on IPSEC tunnel?

[Tim Pushor]

May be a dumb question, but how do I look at traffic going over an IPSEC 
tunnel, on one of the OpenBSD machines? I've tried tcpdump -i enc0 but 
get nothing ..

Re: watch traffic on IPSEC tunnel?

[Tim Pushor]

That was it, thank you :) Its been one of those days :)

Jason Dixon wrote:

On Feb 8, 2007, at 5:15 PM, Tim Pushor wrote:

May be a dumb question, but how do I look at traffic going over an 
IPSEC tunnel, on one of the OpenBSD machines? I've tried tcpdump -i 
enc0 but get nothing ..


The enc0 interface is down by default.  Try bringing it up first 
(ifconfig enc0 up), then run your tcpdump.

Strange vpn trouble

[Tim Pushor]

Hi friends,

I am having a strange problem with a VPN that I've set up between an 
OpenBSD 3.9 server and a Checkpoint VPN-1 device. I've pretty much 
followed the guide at http://anubis.dweebsoft.com/HOWTO/isakmpd.html. I 
have to admit that I don't know enough about ipsec / isakmp.


I do get some errors in the logfile:

Feb  2 05:17:45 fw1 isakmpd[8492]: message_parse_payloads: invalid next 
payload type Unknown 60 in payload of type 8
Feb  2 05:17:45 fw1 isakmpd[8492]: dropped message from 142.59.85.18 
port 500 due to notification type INVALID_PAYLOAD_TYPE
Feb  2 05:17:46 fw1 isakmpd[8492]: message_parse_payloads: invalid next 
payload type Unknown 60 in payload of type 8
Feb  2 05:17:46 fw1 isakmpd[8492]: dropped message from 142.59.85.18 
port 500 due to notification type INVALID_PAYLOAD_TYPE
Feb  2 05:18:08 fw1 isakmpd[8492]: message_parse_payloads: reserved 
field non-zero: 1c
Feb  2 05:18:08 fw1 isakmpd[8492]: dropped message from 142.59.85.18 
port 500 due to notification type PAYLOAD_MALFORMED


But the vpn seems to work. The weird problem I am having is that every 
so often, something strange happens and full packets don't seem to get 
through. Pings still get through, but when cranking up the packet size 
(with ping), once it its 1307, they stop. After an amount of time, 
things resume - and pings 1307+ get through again (and normal data 
starts flowing).


This machine also routes between vlans and I havn't noticed any 
weirdness, although I am going to verify this.


I'm really throwing this out because I don't know where to look. So far 
I've been focused on the key exchange but I'm starting to wonder if 
maybe its somewhere else. If anyone has a clue, I would REALLY 
appreciate it :)


Thanks all,
Tim

Re: Moving a 100GB directory tree with lots of hardlinks

[Tim Pushor]

Have you tried using cpio in passthrough mode? I've used CPIO on big 
systems before with success, although admittedly not on OpenBSD ..


Matthias Bertschy wrote:

OpenBSD 3.7 - i386
Pentium 4 3GHz - 1GB RAM - 2GB swap

Hello list,

For the past 3 weeks, I have been working on a difficult problem: 
moving a backuppc (http://backuppc.sourceforge.net/) pool from a RAID0 
to a big disk, in order to free the RAID0 before rebuilding a RAID5.


The RAID0 has one partition, its size is 2112984700 blocks 
(512-blocks), roughly 1008GB, which is close to the maximum allowed by 
ffs. The big disk is 300GB.


I need to move 96GB of data which are, due to backuppc design, full of 
hardlinks!


So far, I have tried to use:
   1) dd: impossible because the partitions cannot be the same size 
(and the RAID5 won't be the same size as the RAID0)
   2) pax -rw: after transferring almost 70GB, it bails out with a 
Segmentation fault
   3) tar to archive: after something like 60GB, it complains with 
some file name too long errors
   4) gtar to archive (from package gtar-1.15.1p0-static.tgz): ends up 
with a gtar: memory exhauted error

   5) dump to file: successful but
   5') restore from file: stops even before starting due to a no 
memory for entry table error (there is still a lot of unused memory 
and swap - and no ulimit)


Any help is appreciated because I really don't know what to do next.

Matthias Bertschy
Echo Technologies SA

Re: Openbsd 3.9 + trunk

[Tim Pushor]

Steve Glaus wrote:


Ok,  I gotcha, trunk just looked like a ready mad solution for what I 
was trying to do... Could you tell me WHY it's not able to be used for 
that and what it is for?
I've gone the pf route before to but it seems to add a lot of 
complexity to my ruleset


trunk(4) is mainly used to provide redundancy or performance enhancement 
on the same network. I was using it to provide switch redundancy by 
putting one cable in one switch, one in the other, and the switches 
connected together. If I lose a switch, it keeps chugging along.

Re: Openbsd 3.9 + trunk

[Tim Pushor]

Steve Glaus wrote:

Tim Pushor wrote:

Steve Glaus wrote:


Ok,  I gotcha, trunk just looked like a ready mad solution for what 
I was trying to do... Could you tell me WHY it's not able to be used 
for that and what it is for?
I've gone the pf route before to but it seems to add a lot of 
complexity to my ruleset


trunk(4) is mainly used to provide redundancy or performance 
enhancement on the same network. I was using it to provide switch 
redundancy by putting one cable in one switch, one in the other, and 
the switches connected together. If I lose a switch, it keeps 
chugging along.



Alright. Just so I understand.. COULD it be used to do what I'm trying 
to do? When you trunk two network interfaces together, are they 
adressless? Do the devices on the switch address the IP of the pseudo 
trunk interface?
I don't know. I suspect you'd have routing issues, at least. Yes, other 
devices address the IP of the trunk interface, and my real interfaces 
are address-less.

Re: Website(s) being blocked by CARP/PF firewall

[Tim Pushor]

Again, does anyone have any ideas? Can other people access ticketmaster
through their CARP'd NAT firewall?
  
Yeah it works fine over here. How about cranking PF's debugging and 
watching syslog? pfctl -x loud


Tim

Re: Hosting DNS from an openbsd cluster

[Tim Pushor]

Travers Buda wrote:

Hi Friends,

I am wondering anyone can think of why I shouldn't provide secondary
DNS services from a carp cluster of OpenBSD systems? I have an issue
where my primary DNS server is non-redundant, and trying to find a
good place for a secondary. I have a cluster of OpenBSD machines
acting as a router/firewall and would be real convenient to put it
there.

I'd like it to respond to queries on the carp address ..

Can anyone think of a reason to not do this?

Thanks,
Tim




You could use carp, but easier redundancy is already built into the DNS
system. Look into a slave DNS server.

Travers Buda
  
Sorry, I should have been more clear. I am looking for a good spot on my 
network to put a secondary/slave DNS, and I already have a cluster of 
OpenBSD machines acting as a router/firewall and was wondering if there 
was any reason why not to use those as as slaves, since they are already 
redundant and highly available.


Only question is to whether or not to use the/a carp address for the DNS.

Thanks,
Tim

Re: Hosting DNS from an openbsd cluster

[Tim Pushor]

Joachim Schipper wrote:

It will work, but as noted, there's no particular reason to do this;
redundancy is built into the DNS protocol.

  

Well, there is a reason since I need another box to act as a secondary ;-)

The only caveat I can think of is that running services on a firewall
weakens your perimeter security.

  
I concur. In this sealed environment it isn't nearly as much of a 
concern. The box is a router, with a very simple ruleset to allow remote 
administration over the Internet - thats the only real internet traffic.

Finally, don't sync master and CARP - sync master and slave(s) directly.
But that should be obvious.

  
Yeah I thought that. I am still wondering if I should add the carp 
address for the secondary DNS (on the servers resolv.conf), or add 
secondary and tertiary addresses being the primary and backup router ...


Regardless, I think you guys have answered my question. Thanks!

Tim

Joachim

Re: Redundant ethernet Carp (was Re:Soekris)

[Tim Pushor]

Hi Joachim,

Joachim Schipper wrote:

On Thu, Aug 03, 2006 at 02:26:40PM -0600, Tim Pushor wrote:
  
Well, after playing a little with trunk(4), etherchannel, and carp I am 
wondering something:


Trying to achieve both firewall redundancy (via carp) and ethernet 
redundancy (via trunk(4)), would it be possible and (and maybe even 
recommended) to have firewall-1 connected solely to switch-1 and 
firewall-2 connected solely to switch-2, forgo the trunk(4), and just 
use carp to detect if either of the switches has failed, and fail over 
to the other switch/firewall combo?


Am I making sense?



I'm not entirely sure what you intend to achieve, but carp doesn't cross
switches (it works on the local Ethernet segment).
  
Really? I guess I don't understand enough about how carp works. I didn't 
see that as a limitation in any documentation that I read. Why exactly 
is this?


Thanks,
Tim

Redundant ethernet Carp (was Re:Soekris)

[Tim Pushor]

Well, after playing a little with trunk(4), etherchannel, and carp I am 
wondering something:


Trying to achieve both firewall redundancy (via carp) and ethernet 
redundancy (via trunk(4)), would it be possible and (and maybe even 
recommended) to have firewall-1 connected solely to switch-1 and 
firewall-2 connected solely to switch-2, forgo the trunk(4), and just 
use carp to detect if either of the switches has failed, and fail over 
to the other switch/firewall combo?


Am I making sense?

Thanks,
Tim

Re: Soekris

[Tim Pushor]

Jason Dixon wrote:

On Aug 1, 2006, at 2:48 PM, Tim Pushor wrote:

Can anyone recommend another 4 port 10/100 ethernet card that will 
work well with OpenBSD 3.9?


I don't have any recommendations on 4 port cards.  If you have a 
switch that will support it, you should consider using VLANs with a 
gigabit card instead.
Hmm now that is VERY interesting. Would it be possible to run a trunk on 
a vlan, then a carp on the trunk?


Also, why the gigabit? Strictly performance?

Thanks,
Tim

Re: Soekris

[Tim Pushor]

Hi Jason,

Jason Dixon wrote:

On Aug 1, 2006, at 3:13 PM, Tim Pushor wrote:


Jason Dixon wrote:

On Aug 1, 2006, at 2:48 PM, Tim Pushor wrote:

Can anyone recommend another 4 port 10/100 ethernet card that will 
work well with OpenBSD 3.9?


I don't have any recommendations on 4 port cards.  If you have a 
switch that will support it, you should consider using VLANs with a 
gigabit card instead.
Hmm now that is VERY interesting. Would it be possible to run a trunk 
on a vlan, then a carp on the trunk?


Also, why the gigabit? Strictly performance?


I think you're getting your technologies confused.  If you're 
referring to an OpenBSD trunk (versus a Cisco trunk), that is an 
aggregation of physical ports on a switch.  Theoretically, you would 
do this, then layer vlan interfaces on top of the trunk.  However, you 
mentioned that you wanted 4 10/100 interfaces.  Using a single gigabit 
port would enable you to exceed the capacity of 4 10/100 interfaces 
with a single port.  You would have a single physical interface (say, 
em0) connected to a switch port enabled for VLANs (e.g., Cisco 
trunk).  Then you can split up the networks by VLAN, rather than by 
physical connection.  Here's a sample setup.


Thanks a lot for replying. I am new to ethernet redudancy (and carp to 
boot) so I probably don't know what I'm talking about. We have a rack of 
servers that are now for the most part fully redundant. You can pull the 
plug on any box and nothing stops - almost. The ethernet switch is the 
last holdout.


So now I am looking at adding ethernet redundancy to an already 
redundant firewall setup. So now instead of having 1 interface in, 1 
interface out, and 1 interface pfsync, classically I'd need another 2 
ports per server for redundancy. So I was thinking that instead of all 
this, I could run it all on vlans, if openbsd will do it.


Am I wrong in thinking that I'd use a trunk(4) for a redundant ethernet 
connection? If not, then I was thinking that since the servers already 
have 2 gigabit ports on them (bge, from dell 850's) that I could run two 
vlan's each (one internal, one external), trunk(4) the vlans, then 
carp(4) the trunks.. Does that make sense? I'd still use the add-in card 
for pfsync.


(I tried, but I suck at complex ascii art network diagrams)

But to answer your question... no, it is not strictly a performance 
(higher throughput, fewer interrupts, etc) boost.  Having less 
hardware means less opportunity for something to fail (ports, cables, 
etc).

Gotcha. Thanks..

Tim

Re: Soekris

[Tim Pushor]

Stuart Henderson wrote:

The vlan idea makes a fair bit of sense - carp(4) over vlan(4)
over trunk(4) over $some_nic(4) or some other mix - but if this
is used for security be aware that your switch then becomes a
security device. Google will find more information, including
http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a008013159f.shtml

  
Thinking about it more, isn't it carp over trunk over vlan over nic? I'm 
gonna give it a shot here in the next day or so.


I agree about the security issue. The alternative is to have 4 switches 
(for a redundant setup). Since we currently only have 4 (cheap) servers 
in the rack, its a tough sell to suggest that we need as many switches 
as servers, for simple Internet access.


I am going to work on whether its even possible, then review that 
document (thanks for the link BTW) before making any final decisions.


Thank you all for your help!
Tim

(I can post a summary with my conclusion if anyone is interested)

Re: Soekris

[Tim Pushor]

Jason Dixon wrote:

On Aug 1, 2006, at 5:23 PM, Tim Pushor wrote:


Stuart Henderson wrote:

The vlan idea makes a fair bit of sense - carp(4) over vlan(4)
over trunk(4) over $some_nic(4) or some other mix - but if this
is used for security be aware that your switch then becomes a
security device. Google will find more information, including
http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a008013159f.shtml 



Thinking about it more, isn't it carp over trunk over vlan over nic? 
I'm gonna give it a shot here in the next day or so.


No.  The OpenBSD trunk device is for link aggregation and failover, 
both properties of the physical layer.  The OpenBSD vlan device 
provides access to VLAN segments (802.1Q), properties of the data-link 
and network layers.


Again, make sure you're not confusing vendor terminologies here.  An 
OpenBSD trunk is what many vendors refer to as teaming or bonding 
(some do refer to it as trunking).  However, many(?) vendors refer to 
a trunk as a port carrying multiple tagged VLANs.

Hi again Jason,

Then I  must have it wrong. I'll try to clarify:

I have two boxes, with 3 interfaces total each. One interface goes to 
the other box for pfsync. That leaves 2 interfaces each. One interface 
will go to Ethernet switch 1, and one will go to Ethernet switch 2. Each 
interface will be split into 2 vlans, an internal and an external. Now I 
can team these vlan's together for redundancy.


Perhaps I am thinking about this wrong .. Are you suggesting that I 
create a team of physical nics (using trunk(4)), then run vlans over that?
Yes, I realize that trunk is used to refer to more than one thing. I 
have always thought of it though as teaming, but calling it trunk(4) as 
thats how it is documented in OpenBSD.


And a huge thank you to all helping.

Thanks,
Tim

Re: Carp/Pfsync problem

[Tim Pushor]

Sorry to bump this thread, but I'd really like to know how to 
troubleshoot something like this.


Should this work? Should I expect the firewall to fail over a TCP 
session? I'm thinking yes, since it does what its supposed to when 
shutting down the active firewall mid-stream, but not when I pull the 
plug on one.


Thanks again,
Tim

Tim Pushor wrote:

Hi friends,

I am trying to setup my first firewall w/failover via carp  pfsync. I 
have it almost working, but am  having a couple issues. I am hoping 
someone will be able to help :)


First, before I enabled preemption I almost always had one machine 
being master for one of the carp interfaces, and slave for the other 
two. It seemed to work, but just looked troublesome. Enabling 
preemption seemed to solve this. Does this point to a bigger problem 
somewhere?


Second, and what I am really trying to fix - is to have an in progress 
TCP session fail over to the  second firewall. The connection stalls 
and eventually times out when failing over, but attempting to  
re-establish after the failover works (through the second firewall). 
I've confirmed (at least in my  mind) that state updates are being 
properly propagated to the second firewall by watching the pfsync  
interface, and noting the state via pfctl -s state. I've watched 
syslog with pfctl -x loud and didn't  see anything.


Any hints on how I can go about troubleshooting this further? I've 
included as much info as I can think  of. The included PF ruleset is 
just a proof of concept - I realize theres quite a bit more to be 
done,  I'm just trying to get the failover working.


Thanks!,
Tim

BTW If there is any OpenBSD guru in Calgary thats looking for a few 
hours of consultancy I'd love to  hear from you :)


Details:

Both systems are Dell 850 servers w/added Intel Etherexpress Pro 
10/100 cards as the pfsync interface,  with a crossover cable between 
them. OS is OpenBSD 3.9, GENERIC Kernel.


   192.168.1.246
+--+
| Test Workstation |
+--|
 |
+| carp1  |+
|   192.168.1.22   |
|  |
+| carp2  |+
 |  192.168.1.23  |
 ||
192.168.1.20  bge0||bge0 192.168.1.21   
  +-+  +-+

  | fw1 |-fxp0fxp0-| fw2 |
  +-+  +-+
10.0.10.253   bge1||bge1 10.0.10.254
 ||
  ---+---  carp0   ---+---
 10.0.10.1
 |
 |
  +-+
  | Test Server |
  +-+
10.0.10.42

(fw1 fxp0 - 192.168.254.253)
(fs2 fxp0 - 192.168.254.254)


 fw1:

# cat hostname.bge0
inet 192.168.1.20 255.255.255.0 NONE

# cat hostname.bge1
inet 10.0.10.253 255.255.255.0 NONE

# cat hostname.fxp0
inet 192.168.254.253 255.255.255.0 NONE

# cat hostname.carp0
inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 carpdev bge1

# cat hostname.carp1
inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 carpdev 
bge0


# cat hostname.carp2
inet 192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 carpdev 
bge0


# cat hostname.pfsync0
up syncif fxp0

# sysctl -a | grep carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=0
net.inet.carp.arpbalance=0

 fw2:

# cat hostname.bge0
inet 192.168.1.21 255.255.255.0 NONE

# cat hostname.bge1
inet 10.0.10.254 255.255.255.0 NONE

# cat hostname.fxp0
inet 192.168.254.254 255.255.255.0 NONE

# cat hostname.carp0
inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 advskew 128 
carpdev bge1


# cat hostname.carp1
inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 advskew 
128 carpdev bge0


# cat hostname.carp2
192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 advskew 128 
carpdev bge0


# cat hostname.pfsync0
up syncif fxp0

# sysctl -a | grep carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=0
net.inet.carp.arpbalance=0


 PF Rules (identical on both machines)

# cat /etc/pf.conf
ext_if=bge0
int_if=bge1
pfsync_if=fxp0

# All interfaces (real + virtual via carp) thought of as external
ext_ifs={ bge0, carp1, carp2 }

# Our internal network(s). Used for access rules and NAT
internal_nets=10.0.10.0/24

# Define NAT source port range (all source ports will be rewritten to use
# this range)
nat_port_range=20001:65535

# Define virtual carp interface that should be used as NAT source
# (i.e. outbound hide nat will appear to come from this virtual 
interface)

nat_carp=carp1

# real interfaces that have virtual carp

Carp/Pfsync problem

[Tim Pushor]

Hi friends,

I am trying to setup my first firewall w/failover via carp  pfsync. I 
have it almost working, but am  having a couple issues. I am hoping 
someone will be able to help :)


First, before I enabled preemption I almost always had one machine being 
master for one of the carp interfaces, and slave for the other two. It 
seemed to work, but just looked troublesome. Enabling preemption seemed 
to solve this. Does this point to a bigger problem somewhere?


Second, and what I am really trying to fix - is to have an in progress 
TCP session fail over to the  second firewall. The connection stalls and 
eventually times out when failing over, but attempting to  re-establish 
after the failover works (through the second firewall). I've confirmed 
(at least in my  mind) that state updates are being properly propagated 
to the second firewall by watching the pfsync  interface, and noting the 
state via pfctl -s state. I've watched syslog with pfctl -x loud and 
didn't  see anything.


Any hints on how I can go about troubleshooting this further? I've 
included as much info as I can think  of. The included PF ruleset is 
just a proof of concept - I realize theres quite a bit more to be done,  
I'm just trying to get the failover working.


Thanks!,
Tim

BTW If there is any OpenBSD guru in Calgary thats looking for a few 
hours of consultancy I'd love to  hear from you :)


Details:

Both systems are Dell 850 servers w/added Intel Etherexpress Pro 10/100 
cards as the pfsync interface,  with a crossover cable between them. OS 
is OpenBSD 3.9, GENERIC Kernel.


   192.168.1.246
+--+
| Test Workstation |
+--|
 |
+| carp1  |+
|   192.168.1.22   |
|  |
+| carp2  |+
 |  192.168.1.23  |
 ||
192.168.1.20  bge0||bge0 192.168.1.21   
  +-+  +-+

  | fw1 |-fxp0fxp0-| fw2 |
  +-+  +-+
10.0.10.253   bge1||bge1 10.0.10.254
 ||
  ---+---  carp0   ---+---
 10.0.10.1
 |
 |
  +-+
  | Test Server |
  +-+
10.0.10.42

(fw1 fxp0 - 192.168.254.253)
(fs2 fxp0 - 192.168.254.254)


 fw1:

# cat hostname.bge0
inet 192.168.1.20 255.255.255.0 NONE

# cat hostname.bge1
inet 10.0.10.253 255.255.255.0 NONE

# cat hostname.fxp0
inet 192.168.254.253 255.255.255.0 NONE

# cat hostname.carp0
inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 carpdev bge1

# cat hostname.carp1
inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 carpdev bge0

# cat hostname.carp2
inet 192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 carpdev bge0

# cat hostname.pfsync0
up syncif fxp0

# sysctl -a | grep carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=0
net.inet.carp.arpbalance=0

 fw2:

# cat hostname.bge0
inet 192.168.1.21 255.255.255.0 NONE

# cat hostname.bge1
inet 10.0.10.254 255.255.255.0 NONE

# cat hostname.fxp0
inet 192.168.254.254 255.255.255.0 NONE

# cat hostname.carp0
inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 advskew 128 
carpdev bge1


# cat hostname.carp1
inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 advskew 
128 carpdev bge0


# cat hostname.carp2
192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 advskew 128 
carpdev bge0


# cat hostname.pfsync0
up syncif fxp0

# sysctl -a | grep carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=0
net.inet.carp.arpbalance=0


 PF Rules (identical on both machines)

# cat /etc/pf.conf
ext_if=bge0
int_if=bge1
pfsync_if=fxp0

# All interfaces (real + virtual via carp) thought of as external
ext_ifs={ bge0, carp1, carp2 }

# Our internal network(s). Used for access rules and NAT
internal_nets=10.0.10.0/24

# Define NAT source port range (all source ports will be rewritten to use
# this range)
nat_port_range=20001:65535

# Define virtual carp interface that should be used as NAT source
# (i.e. outbound hide nat will appear to come from this virtual interface)
nat_carp=carp1

# real interfaces that have virtual carp addresses associated with them
carp_interfaces={ bge0, bge1 }

# Test internal HTTP server
tstsrv_ext=192.168.1.22
tstsrv_int=10.0.10.42
tstsrv_port=80

###
### NAT
###

# Provide 'hide mode' nat for the entire subnet

nat on $ext_if from $internal_nets to any - $nat_carp port $nat_port_range

# Test HTTP access

rdr on $ext_if proto tcp from any to $tstsrv_ext 

Re: Carp/Pfsync problem

[Tim Pushor]

Thanks Kian,

That didn't have an effect. I suspected that it woudn't as I am getting 
state information, and ifconfig listed fxp0 as the syncdev even though I 
had syncif in the hostname file.


As for the multiple carp addresses - This is in a lab environment but 
will end up protecting a rack of machines in a colo. I'm planning on 
having a carp address for each external address that's required (not 
many - maybe 4-5 eventually).


Thanks,
Tim

Kian Mohageri wrote:

Change 'syncif' to 'syncdev' in your hostname.pfsync files.

Also, out of curiosity, why are there two CARP addresses between the
workstation and firewalls?

Kian

On 9/20/06, Tim Pushor [EMAIL PROTECTED] wrote:
  

Hi friends,

I am trying to setup my first firewall w/failover via carp  pfsync. I
have it almost working, but am  having a couple issues. I am hoping
someone will be able to help :)

First, before I enabled preemption I almost always had one machine being
master for one of the carp interfaces, and slave for the other two. It
seemed to work, but just looked troublesome. Enabling preemption seemed
to solve this. Does this point to a bigger problem somewhere?

Second, and what I am really trying to fix - is to have an in progress
TCP session fail over to the  second firewall. The connection stalls and
eventually times out when failing over, but attempting to  re-establish
after the failover works (through the second firewall). I've confirmed
(at least in my  mind) that state updates are being properly propagated
to the second firewall by watching the pfsync  interface, and noting the
state via pfctl -s state. I've watched syslog with pfctl -x loud and
didn't  see anything.

Any hints on how I can go about troubleshooting this further? I've
included as much info as I can think  of. The included PF ruleset is
just a proof of concept - I realize theres quite a bit more to be done,
I'm just trying to get the failover working.

Thanks!,
Tim

BTW If there is any OpenBSD guru in Calgary thats looking for a few
hours of consultancy I'd love to  hear from you :)

Details:

Both systems are Dell 850 servers w/added Intel Etherexpress Pro 10/100
cards as the pfsync interface,  with a crossover cable between them. OS
is OpenBSD 3.9, GENERIC Kernel.

192.168.1.246
 +--+
 | Test Workstation |
 +--|
  |
 +| carp1  |+
 |   192.168.1.22   |
 |  |
 +| carp2  |+
  |  192.168.1.23  |
  ||
192.168.1.20  bge0||bge0 192.168.1.21
   +-+  +-+
   | fw1 |-fxp0fxp0-| fw2 |
   +-+  +-+
10.0.10.253   bge1||bge1 10.0.10.254
  ||
   ---+---  carp0   ---+---
  10.0.10.1
  |
  |
   +-+
   | Test Server |
   +-+
 10.0.10.42

(fw1 fxp0 - 192.168.254.253)
(fs2 fxp0 - 192.168.254.254)


 fw1:

# cat hostname.bge0
inet 192.168.1.20 255.255.255.0 NONE

# cat hostname.bge1
inet 10.0.10.253 255.255.255.0 NONE

# cat hostname.fxp0
inet 192.168.254.253 255.255.255.0 NONE

# cat hostname.carp0
inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 carpdev bge1

# cat hostname.carp1
inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 carpdev
bge0

# cat hostname.carp2
inet 192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 carpdev
bge0

# cat hostname.pfsync0
up syncif fxp0

# sysctl -a | grep carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=0
net.inet.carp.arpbalance=0

 fw2:

# cat hostname.bge0
inet 192.168.1.21 255.255.255.0 NONE

# cat hostname.bge1
inet 10.0.10.254 255.255.255.0 NONE

# cat hostname.fxp0
inet 192.168.254.254 255.255.255.0 NONE

# cat hostname.carp0
inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 advskew 128
carpdev bge1

# cat hostname.carp1
inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 advskew
128 carpdev bge0

# cat hostname.carp2
192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 advskew 128
carpdev bge0

# cat hostname.pfsync0
up syncif fxp0

# sysctl -a | grep carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=0
net.inet.carp.arpbalance=0


 PF Rules (identical on both machines)

# cat /etc/pf.conf
ext_if=bge0
int_if=bge1
pfsync_if=fxp0

# All interfaces (real + virtual via carp) thought of as external
ext_ifs={ bge0, carp1, carp2 }

# Our internal network(s). Used for access rules and NAT
internal_nets=10.0.10.0/24