This config works for me:
OpenBSD 4.3 as GW and Debian Linux with OpenSWAN as client, and
the package ike is installed under Linux, too.
OpenBSD:
ike esp from any to 172.16.1.98 quick auth hmac-sha1 enc aes
group modp1024 psk "IMTEHLINUXCLIENT"
Linux:
/etc/ipsec.conf
version 2.0
cono,g setup
interfaces=wlan0
plutodebug=ballb
nat traversal=yes
plutowait=yes
nhelpers=0
uniqueids=yes
conn openbsd
type=transport
left=172.16.1.98
right=172.16.1.1
rightsubnet=0.0.0.0/0
keyexchange=ike
esp=aes128-sha1
ike=aes128-sha1-modp1024
auto=route
auth=esp
authby=secret
pfs=yes
keyingtries=rekeymargin=4m
disablearrivalcheck=no
rekey=yes
aggrmode=no
/etc/ipsec.secrets
172.16.1.1 172.16.1.98: PSK "IMTEHLINUXCLIENT"
Laurent CARON wrote:
> Sean Malloy wrote:
>> It looks like you are trying to use different encryption algorithms and
>> hash functions for the phase 2 SA. They need to match at both end points.
>> It looks like the Linux box is configured to do 3DES and SHA1. The
>> OpenBSD box is configured to do AES and SHA256.
>>
>
> Hi,
>
> Even with this setup it doesn't establish the tunnel properly.
>
> BSD:
>
>
> ike esp from 10.50.0.0/24 to 192.168.9.0/24 peer LINUX_PUBLIC \
> main auth hmac-sha1 enc aes group modp1024 \
> quick auth hmac-sha1 enc aes group modp1024 \
> psk "passphrase"
>
> Linux:
>
> conn lnx-bsd
> left=LINUX_PUBLIC
> leftsubnet=192.168.9.0/24
> right=BSD_PUBLIC
> rightsubnet=10.50.0.0/24
> authby=secret
> ike=aes-sha1-modp1024
> auto=start
> esp=aes-sha1
> keyexchange=ike
>
> Here are the logs:
>
> Linux side:
>
> "lnx-bsd" #2: I did not send a certificate because I do not have one.
> "lnx-bsd" #2: transition from state STATE_MAIN_I2 to state STATE_MAIN_I3
> "lnx-bsd" #2: STATE_MAIN_I3: sent MI3, expecting MR3
> "lnx-bsd" #7: responding to Main Mode
> "lnx-bsd" #7: transition from state STATE_MAIN_R0 to state STATE_MAIN_R1
> "lnx-bsd" #7: STATE_MAIN_R1: sent MR1, expecting MI2
> "lnx-bsd" #7: transition from state STATE_MAIN_R1 to state STATE_MAIN_R2
> "lnx-bsd" #7: STATE_MAIN_R2: sent MR2, expecting MI3
> "lnx-bsd" #7: ignoring informational payload, type IPSEC_INITIAL_CONTACT
> "lnx-bsd" #7: Main mode peer ID is ID_IPV4_ADDR: 'BSD_PUBLIC'
> "lnx-bsd" #7: I did not send a certificate because I do not have one.
> "lnx-bsd" #7: transition from state STATE_MAIN_R2 to state STATE_MAIN_R3
> "lnx-bsd" #7: STATE_MAIN_R3: sent MR3, ISAKMP SA established
> {auth=OAKLEY_PRESHARED_KEY cipher=aes_128 prf=oakley_sha group=modp1024}
> "lnx-bsd" #8: responding to Quick Mode {msgid:de9df09c}
> "lnx-bsd" #8: transition from state STATE_QUICK_R0 to state STATE_QUICK_R1
> "lnx-bsd" #8: STATE_QUICK_R1: sent QR1, inbound IPsec SA installed,
> expecting QI2
> "lnx-bsd" #8: transition from state STATE_QUICK_R1 to state STATE_QUICK_R2
> "lnx-bsd" #8: STATE_QUICK_R2: IPsec SA established {ESP=>0xa2b93f3b
> <0x38351b1b xfrm=AES_128-HMAC_SHA1 NATD=none DPD=none}
>
>
> BSD side:
> Aug 27 10:17:52 fw-001 isakmpd[11393]: message_parse_payloads: invalid
> next payload type <Unknown 96> in payload of type 5
> Aug 27 10:17:52 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type INVALID_PAYLOAD_TYPE
> Aug 27 10:18:03 fw-001 isakmpd[11393]: message_parse_payloads: invalid
> next payload type <Unknown 96> in payload of type 5
> Aug 27 10:18:03 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type INVALID_PAYLOAD_TYPE
> Aug 27 10:18:22 fw-001 isakmpd[11393]: message_parse_payloads: invalid
> next payload type <Unknown 96> in payload of type 5
> Aug 27 10:18:22 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type INVALID_PAYLOAD_TYPE
> Aug 27 10:20:14 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: d
> Aug 27 10:20:14 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:20:24 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: d
> Aug 27 10:20:24 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:20:44 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: d
> Aug 27 10:20:44 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:22:34 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 6d
> Aug 27 10:22:34 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:22:45 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 6d
> Aug 27 10:22:45 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:23:04 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 6d
> Aug 27 10:23:04 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:24:56 fw-001 isakmpd[11393]: message_parse_payloads: invalid
> next payload type <Unknown 45> in payload of type 5
> Aug 27 10:24:56 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type INVALID_PAYLOAD_TYPE
> Aug 27 10:25:05 fw-001 isakmpd[11393]: message_parse_payloads: invalid
> next payload type <Unknown 45> in payload of type 5
> Aug 27 10:25:05 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type INVALID_PAYLOAD_TYPE
> Aug 27 10:25:26 fw-001 isakmpd[11393]: message_parse_payloads: invalid
> next payload type <Unknown 45> in payload of type 5
> Aug 27 10:25:26 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type INVALID_PAYLOAD_TYPE
> Aug 27 10:27:16 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: b0
> Aug 27 10:27:16 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:27:27 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: b0
> Aug 27 10:27:27 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:27:47 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: b0
> Aug 27 10:27:47 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:29:38 fw-001 isakmpd[11393]: message_parse_payloads: invalid
> next payload type <Unknown 42> in payload of type 5
> Aug 27 10:29:38 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type INVALID_PAYLOAD_TYPE
> Aug 27 10:29:48 fw-001 isakmpd[11393]: message_parse_payloads: invalid
> next payload type <Unknown 42> in payload of type 5
> Aug 27 10:29:48 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type INVALID_PAYLOAD_TYPE
> Aug 27 10:30:07 fw-001 isakmpd[11393]: message_parse_payloads: invalid
> next payload type <Unknown 42> in payload of type 5
> Aug 27 10:30:07 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type INVALID_PAYLOAD_TYPE
> Aug 27 10:31:59 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 8
> Aug 27 10:31:59 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:32:08 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 8
> Aug 27 10:32:08 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:32:29 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 8
> Aug 27 10:32:29 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:34:19 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 17
> Aug 27 10:34:19 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:34:30 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 17
> Aug 27 10:34:30 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:34:49 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 17
> Aug 27 10:34:49 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:36:41 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 64
> Aug 27 10:36:41 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:36:50 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 64
> Aug 27 10:36:50 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
> Aug 27 10:37:11 fw-001 isakmpd[11393]: message_parse_payloads: reserved
> field non-zero: 64
> Aug 27 10:37:11 fw-001 isakmpd[11393]: dropped message from PUBLIC_LINUX
> port 500 due to notification type PAYLOAD_MALFORMED
>
> Thanks
>
> Laurent