Re: Experience with isakmpd/ipsec in production?
Hi, On Mon, 21.08.2006 at 10:23:43 -0400, Melameth, Daniel D. [EMAIL PROTECTED] wrote: We have since changed how we're doing this, but we had a Cisco and OpenBSD VPN running for a few years. why, and how did you change? What's better now? Best, --Toni++
Re: Experience with isakmpd/ipsec in production?
Hi, On Mon, 21.08.2006 at 15:43:14 +0200, Sven Ingebrigt Ulland [EMAIL PROTECTED] wrote: How long have you been running openbsd isakmpd/ipsec (in production)? I think I run this stuff since around 2000, or 2001 at the latest. What problems, if any, have you had with the openbsd vpn implementations? Which of them are the most recurring? How do you usually fix them? There were some compatibility issues in earlier releases which were fixed quite fast (MANY thanks!). We had a few cases where isakmpd went down, but decided early to fix these by using process supervisors (also years ago, I don't know if these problems are still there). We use almost all easy features isakmpd has. Otherwise, I can't remember a problem. Have you experienced any interoperability problems when establishing tunnels with peers that run other implementations (cisco, checkpoint, etc)? And if so, how do you work around those? My experience is that most other devices I encountered so far are much less flexible and powerful than is OpenBSD. So, interoperating often means finding out what the other side can't, and then take the best out of what remains. I can only say that OpenBSD is very much recommended for serious IPSEC usage. Best, --Toni++
Re: Experience with isakmpd/ipsec in production?
Sven Ingebrigt Ulland wrote:
[...]
Thanks to all of you who have contributed with your
experiences with isakmpd/ipsec in OpenBSD. After some time
now, I've seen some more of the good and bad sides of our
VPN setup, and I'll share it with you.
How long have you been running openbsd isakmpd/ipsec (in production)?
It's been running for over a year now, and it's been very
stable.
What problems, if any, have you had with the openbsd vpn
implementations? Which of them are the most recurring? How do you
usually fix them?
There are a few issues that I've seen with the
implementation, or more aptly, my lack of detailed knowledge
of the IPSec specs:
1) isakmpd isn't easily debuggable. When some error occurs,
or when something expected does not occur, it is hard to
know what debug level to increase in isakmpd. Of course, it
would help a great deal to have detailed knowledge of the
IPSec specs here, but I haven't found the time to get to
know them very well. In that respect, I find the man page
for isakmpd to be somewhat lacking.
Not knowing how to debug properly leads to problems
determining on which side the error is located, or if the
fault is in an intermediate network. This can lead to a
blame game with the other side, which doesn't do anyone much
good. About that, I'm interested in hearing of good tips on
debugging stuff like this. I use the normal tools like ping,
{tcp,udp,icmp} traceroute, hping, tcpdump filtering on udp
port 500 || proto 50 and isakmpd logging, but still fall
short of determining the exact cause most of the time. Maybe
I'm using the tools the wrong way.
2) A common problem is that we simply stop seeing data from
one or more peers. (Our endpoint is set up as a slave for
all the connections, so it is our peers that initiate
connections.) What we usually do then, is to dump packets on
the network interface to determine whether the peer is
completely dead or if it's hung.
3) On some occations, the peer is hung up somehow, and keeps
trying to send us an invalid SPI. Our IPSec rejects those,
but it keeps sending them. What we then do is to stop
isakmpd and then start it again. For some reason, this fixes
the problem. We haven't dumped the traffic while restarting
isakmpd yet, but it probably sends some seize and desist
signal to all the peers. I'm wondering if it's possible to
send this signal to just one peer.. that would keep the
other tunnels alive.
Have you experienced any interoperability problems when establishing
tunnels with peers that run other implementations (cisco, checkpoint,
etc)? And if so, how do you work around those?
Our peers mostly run cisco or checkpoint equipment. In the
isakmpd logs we see a *lot* of the following messages:
dropped message from 172.29.9.43 port 500 due to notification type
PAYLOAD_MALFORMED
dropped message from 172.29.9.43 port 500 due to notification type
INVALID_PAYLOAD_TYPE
message_parse_payloads: invalid next payload type Unknown 111 in payload of type
8
(the number 111 varies from ~25 to ~125)
message_parse_payloads: reserved field non-zero: 17
(the number varies from 0x00 to 0xff).
Having a look through the IPSec specs (33 RFCs! Damn, where
to start?) would probably explain some of this behaviour.
I'm guessing the proprietary boxes use some in-house
extensions. Tips are greatly welcome!
regards,
Sven U
Re: Experience with isakmpd/ipsec in production?
On Thu, Oct 05, 2006 at 09:59:27PM +0200, Sven Ulland wrote:
Sven Ingebrigt Ulland wrote:
[...]
Thanks to all of you who have contributed with your
experiences with isakmpd/ipsec in OpenBSD. After some time
now, I've seen some more of the good and bad sides of our
VPN setup, and I'll share it with you.
How long have you been running openbsd isakmpd/ipsec (in production)?
It's been running for over a year now, and it's been very
stable.
What problems, if any, have you had with the openbsd vpn
implementations? Which of them are the most recurring? How do you
usually fix them?
There are a few issues that I've seen with the
implementation, or more aptly, my lack of detailed knowledge
of the IPSec specs:
1) isakmpd isn't easily debuggable. When some error occurs,
or when something expected does not occur, it is hard to
know what debug level to increase in isakmpd. Of course, it
would help a great deal to have detailed knowledge of the
IPSec specs here, but I haven't found the time to get to
know them very well. In that respect, I find the man page
for isakmpd to be somewhat lacking.
Not knowing how to debug properly leads to problems
determining on which side the error is located, or if the
fault is in an intermediate network. This can lead to a
blame game with the other side, which doesn't do anyone much
good. About that, I'm interested in hearing of good tips on
debugging stuff like this. I use the normal tools like ping,
{tcp,udp,icmp} traceroute, hping, tcpdump filtering on udp
port 500 || proto 50 and isakmpd logging, but still fall
short of determining the exact cause most of the time. Maybe
I'm using the tools the wrong way.
2) A common problem is that we simply stop seeing data from
one or more peers. (Our endpoint is set up as a slave for
all the connections, so it is our peers that initiate
connections.) What we usually do then, is to dump packets on
the network interface to determine whether the peer is
completely dead or if it's hung.
3) On some occations, the peer is hung up somehow, and keeps
trying to send us an invalid SPI. Our IPSec rejects those,
but it keeps sending them. What we then do is to stop
isakmpd and then start it again. For some reason, this fixes
the problem. We haven't dumped the traffic while restarting
isakmpd yet, but it probably sends some seize and desist
signal to all the peers. I'm wondering if it's possible to
send this signal to just one peer.. that would keep the
other tunnels alive.
Have you experienced any interoperability problems when establishing
tunnels with peers that run other implementations (cisco, checkpoint,
etc)? And if so, how do you work around those?
Our peers mostly run cisco or checkpoint equipment. In the
isakmpd logs we see a *lot* of the following messages:
dropped message from 172.29.9.43 port 500 due to notification type
PAYLOAD_MALFORMED
dropped message from 172.29.9.43 port 500 due to notification type
INVALID_PAYLOAD_TYPE
message_parse_payloads: invalid next payload type Unknown 111 in payload
of type 8
(the number 111 varies from ~25 to ~125)
message_parse_payloads: reserved field non-zero: 17
(the number varies from 0x00 to 0xff).
Having a look through the IPSec specs (33 RFCs! Damn, where
to start?) would probably explain some of this behaviour.
I'm guessing the proprietary boxes use some in-house
extensions. Tips are greatly welcome!
I found the references to the isakmpd.fifo mentioned in
/usr/src/sbin/isakmpd/DESIGN-NOTES useful for tearing down
specific tunnels. The part I found useful was about 57% through
the file under User control.
Hope this helps with your situation.
Re: Experience with isakmpd/ipsec in production?
On Mon, 2006-08-21 at 15:43 +0200, Sven Ingebrigt Ulland wrote: How long have you been running openbsd isakmpd/ipsec (in production)? We've been using them since 3.9 and got small quirks mostly due to our misunderstanding of protocols and implementations, a little also due to the initial lack of openbsd-standard-level documentation :) Any issue was resolved with a small search on code or mailing list archive or as a last resource asking directly to [EMAIL PROTECTED] Now we got a 10 node VPN lan based totally on -current as of mid of August with more the 70 tunnels in it. I will add 8 more peers during September. So far very happy with reliability and maintenance facility. A small side note, I'm waiting the 'fix' for totally take advantage of Via C3/C7 crypto features and hope they will be in for 4.0 or just a little after :) even if my users are very happy with the current performance. Regards -- Massimo.run();
Re: Experience with isakmpd/ipsec in production?
On Tue, Aug 22, 2006 at 04:10:22PM +0200, Massimo Lusetti wrote: On Mon, 2006-08-21 at 15:43 +0200, Sven Ingebrigt Ulland wrote: snip I'm making heavy usage of VPN to mount NFS over (so there are huge amounts of traffic going over the tunnel at maximum speed the CPUs can handle) and IPSEC itself works very reliable (at least compared to openvpn, which I never had real luck with). The only issue, which remains: I have to reboot ALL clients, which have an active NFS mount after the server went down. But that has nothing to do with IPSEC, thus I shut up about it at this point. A small side note, I'm waiting the 'fix' for totally take advantage of Via C3/C7 crypto features and hope they will be in for 4.0 or just a little after :) even if my users are very happy with the current performance. Is there development going on with the VIA issue? Would be great I'm eager for near-line-speed (100mbit) @25W :) Regards, ahb
Experience with isakmpd/ipsec in production?
We are about to deploy some fairly critical VPN functionality in our network, and for that purpose we're considering using OpenBSD with isakmp/ipsec. We've had a test setup running for some time now with no problems, but I'm interested in hearing about your long-term experiences with running openbsd ipsec/isakmpd in critical production environments. My excuses for the survey-ish feeling of this post. How long have you been running openbsd isakmpd/ipsec (in production)? What problems, if any, have you had with the openbsd vpn implementations? Which of them are the most recurring? How do you usually fix them? Have you experienced any interoperability problems when establishing tunnels with peers that run other implementations (cisco, checkpoint, etc)? And if so, how do you work around those? On the outside, it seems to me that the vpn implementation in openbsd is good and stable, which could also stem from the corporate funding it received. And the relevant files in cvs seem to be changed rather infrequently.. also a good sign. But I'm not familiar with the inside, which is what i was hoping you could help out with. regards, Sven U
Re: Experience with isakmpd/ipsec in production?
Sven Ingebrigt Ulland wrote: We are about to deploy some fairly critical VPN functionality in our network, and for that purpose we're considering using OpenBSD with isakmp/ipsec. We've had a test setup running for some time now with no problems, but I'm interested in hearing about your long-term experiences with running openbsd ipsec/isakmpd in critical production environments. My excuses for the survey-ish feeling of this post. How long have you been running openbsd isakmpd/ipsec (in production)? We have since changed how we're doing this, but we had a Cisco and OpenBSD VPN running for a few years. What problems, if any, have you had with the openbsd vpn implementations? Which of them are the most recurring? How do you usually fix them? We had zero problems--with the exception of a couple rare MTU issues and, while probably not the ideal resolution, fixing the MTU on the affected hosts resolved these. Have you experienced any interoperability problems when establishing tunnels with peers that run other implementations (cisco, checkpoint, etc)? And if so, how do you work around those? None--after finding the correct initial configuration everything just worked and continued to.
Re: Experience with isakmpd/ipsec in production?
Have you experienced any interoperability problems when establishing tunnels with peers that run other implementations (cisco, checkpoint, etc)? And if so, how do you work around those? None--after finding the correct initial configuration everything just worked and continued to. One example of our finding the correct initial configuration when connecting OpenBSD VPN to a SonicWall VPN. http://cisx1.uma.maine.edu/~wbackman/vpn/ Things are a lot more simple now, thanks to ipsecctl.
Re: Experience with isakmpd/ipsec in production?
Sven Ingebrigt Ulland wrote: We are about to deploy some fairly critical VPN functionality in our network, and for that purpose we're considering using OpenBSD with isakmp/ipsec. We've had a test setup running for some time now with no problems, but I'm interested in hearing about your long-term experiences with running openbsd ipsec/isakmpd in critical production environments. My excuses for the survey-ish feeling of this post. How long have you been running openbsd isakmpd/ipsec (in production)? What problems, if any, have you had with the openbsd vpn implementations? Which of them are the most recurring? How do you usually fix them? Have you experienced any interoperability problems when establishing tunnels with peers that run other implementations (cisco, checkpoint, etc)? And if so, how do you work around those? On the outside, it seems to me that the vpn implementation in openbsd is good and stable, which could also stem from the corporate funding it received. And the relevant files in cvs seem to be changed rather infrequently.. also a good sign. But I'm not familiar with the inside, which is what i was hoping you could help out with. regards, Sven U We have been running vpn's here for over a year using isakmpd on OBSD beginning with 3.7. We have currently a mix of 3.7 3.8 and 3.9, on SPARC and AMD, all on SUN hardware. We use it to connect medical system at two county jails to our hospital data center. We also use it to connect pharmacists and radiologists to our systems for after hours service. So an entire county medical infrastructure would be unable to issue meds or read x-rays after hours if our vpn tunnels were down. We have found OBSD to be very reliable. We have a single 'hang' that could not be resolved by HUP-ing isakmpd, so we simply failed over to the sasync secondary system. Otherwise once these puppies go up ... they pretty much just work and work and work. Interop with Checkpoint has been dead simple, with Cisco less so. I have found that when tunneling to something the other side has called Cisco VPN concentrators, things go more smoothly if you use 3DES and MD5. Seems that if you try to use SHA that we never seem to get past a phase one state. One thing about OBSD you will find to be truly bizarre is that things work as documented AND the man pages are concise and useful AND all features and config files are documented. I used to manage a small herd of Checkpoints and Netscreens, I have never looked back.
Re: Experience with isakmpd/ipsec in production?
We have been using OBSD VPN tunnels for a while now, since 3.5 We still have a handful of 3.5 systems running and have no issues. We have approx 35 locations running all the time and the tunnels just work. We also have them connecting to our datacenter which, unfortunally we have not had time yet to get of checkpoint NG. They connect and have zero issues, unless we apply an update to checkpoint and then we have to re-hup isakmpd or wait until the tunnels neg themselves. Usually, we just run a script remotely to do the re-hup for us from a trusted environment and it saves us a lot of time. We only update checkpoint after hours, but doing anything on OBSD is 100% perfect. We are replacing Checkpoint to 3.9 using 2 redundant systems this month.. the back side of our network (datacenter) is gigabit and OBSD is working great for us. Let me say, checkpoint is toast for us, we are huge fans of OBSD isakmpd. Just hoping that SASYNC re-neg on failback works better in the next release of OBSD, but let me say, that is zero impact on our go forward, I love what the OBSD community has put together. Great OS James -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Sven Ingebrigt Ulland Sent: Monday, August 21, 2006 10:43 AM To: misc@openbsd.org Subject: Experience with isakmpd/ipsec in production? We are about to deploy some fairly critical VPN functionality in our network, and for that purpose we're considering using OpenBSD with isakmp/ipsec. We've had a test setup running for some time now with no problems, but I'm interested in hearing about your long-term experiences with running openbsd ipsec/isakmpd in critical production environments. My excuses for the survey-ish feeling of this post. How long have you been running openbsd isakmpd/ipsec (in production)? What problems, if any, have you had with the openbsd vpn implementations? Which of them are the most recurring? How do you usually fix them? Have you experienced any interoperability problems when establishing tunnels with peers that run other implementations (cisco, checkpoint, etc)? And if so, how do you work around those? On the outside, it seems to me that the vpn implementation in openbsd is good and stable, which could also stem from the corporate funding it received. And the relevant files in cvs seem to be changed rather infrequently.. also a good sign. But I'm not familiar with the inside, which is what i was hoping you could help out with. regards, Sven U
