Re: [strongSwan] AWS EC2 IKEv2 tunnel up but no throughput
Hi Noel/users, Does no-one have any suggestions as to what we can try here? I've looked long and hard at this and feel that I have exhausted any obvious settings to adjust. The only thing slightly unusual about the setup is that we are using a single interface, but this has been documented as working so it really should not be an issue. Packets not traversing the tunnel is confirmed by... swanctl --list-sas tunnel1: #1, ESTABLISHED, IKEv2, 88d5c48c82546516_i* a2d86821f1f52625_r local '52.8.104.97' @ 48.138.201.70[4500] remote '68.148.15.170' @ 68.148.15.170[4500] AES_GCM_16-256/PRF_HMAC_SHA2_384/ECP_384 established 582s ago, reauth in 85216s tunnel1: #1, reqid 1, INSTALLED, TUNNEL-in-UDP, ESP:AES_GCM_16-256 installed 582s ago, rekeying in 2218s, expires in 3018s in c675ba46, 0 bytes, 0 packets out 9333b51f, 0 bytes, 0 packets local 48.138.201.64/26 remote 198.168.248.0/2 and ... ipsec listcounters List of IKE counters: ikeInitRekey 0 ikeRspRekey 0 ikeChildSaRekey 0 ikeInInvalid 0 ikeInInvalidSpi 0 ikeInInitReq 0 ikeInInitRsp 2 ikeOutInitReq 2 ikeOutInitRsp 0 ikeInAuthReq 0 ikeInAuthRsp 2 ikeOutAuthReq 2 ikeOutAuthRsp 0 ikeInCrChildReq 0 ikeInCrChildRsp 0 ikeOutCrChildReq 0 ikeOutCrChildRsp 0 ikeInInfoReq200 ikeInInfoRsp 0 ikeOutInfoReq 0 ikeOutInfoRsp 200 Any packets destined to the ip's on the rightsubnet from the left vpn-gw itself do not escape, VPC flow logs confirm that no packets from the vpn-gw destined for the rightsubnet traverse the vpc, and any packets destined for the rightsubnet from the leftsubnet route to the vpng-gw as expected but progress no further. The problem seems to me to be tied in with the xfrm policy but this is speculative. Martians were detected only briefly 3 days ago with two singular identical entries kernel: IPv4: martian source 48.138.201.70 from 68.148.15.170, on dev eth0 All traffic between the VPN endpoints is encapsulated by NAT-T UDP 4500 tcpdump shows checksum errors for packets originating from the left vpn-gw to the rightsubnet, but we do not see any 02:45:59.096217 IP (tos 0x0, ttl 255, id 63930, offset 0, flags [none], proto TCP (6), length 60) 48.138.201.70.39268 > 198.168.248.4.13865: Flags [S], cksum 0xb5e1 (incorrect -> 0x6af2), seq 329764005, win 26883, options [mss 8961,sackOK,TS val 3337366083 ecr 0,nop,wscale 7], length 0 XfrmOutStateModeError count is not incremented in /proc/self/net/xfrm_stat and we have mangle rules in place to clamp to mss to 1360. I could not find any info out there relating to the IKE counters I assume if we were traversing the tunnel successfully that we'd see incremented values in ikeInInfoRsp & ikeOutInfoReq ?? also swanctl --list-sas naturally... Hoping someone can help here. Cheers Lew On Mon, 5 Jul 2021 at 19:54, Lewis Shobbrook wrote: > > Thanks for your reply Noel, > Landed in my spam folder... > I'm testing with a curl to a known endpoint from the vpn gateway > itself and also from the associated local subnet that is accepted on > the other side. > Here's the output of iptables-save which has changed quite a bit over > various efforts to realise throughput. > > # Generated by iptables-save v1.8.4 on Mon Jul 5 09:52:05 2021 > *mangle > :PREROUTING ACCEPT [131559:43589022] > :INPUT ACCEPT [129402:43459544] > :FORWARD ACCEPT [2155:129300] > :OUTPUT ACCEPT [131976:29447953] > :POSTROUTING ACCEPT [134131:29577253] > -A FORWARD -p tcp -m policy --dir in --pol ipsec -m tcp --tcp-flags > SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS --set-mss 1360 > -A FORWARD -p tcp -m policy --dir out --pol ipsec -m tcp --tcp-flags > SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS --set-mss 1360 > COMMIT > # Completed on Mon Jul 5 09:52:05 2021 > # Generated by iptables-save v1.8.4 on Mon Jul 5 09:52:05 2021 > *nat > :PREROUTING ACCEPT [0:0] > :INPUT ACCEPT [0:0] > :OUTPUT ACCEPT [0:0] > :POSTROUTING ACCEPT [0:0] > -A POSTROUTING -m policy --dir out --pol ipsec -j ACCEPT > -A POSTROUTING -s 48.138.201.64/26 -o eth0 -m policy --dir out --pol > ipsec -j ACCEPT > COMMIT > # Completed on Mon Jul 5 09:52:05 2021 > # Generated by iptables-save v1.8.4 on Mon Jul 5 09:52:05 2021 > *filter > :INPUT ACCEPT [42760:14832665] > :FORWARD ACCEPT [774:46440] > :OUTPUT ACCEPT [43602:9751014] > COMMIT > > > Cheers, > > Lew > > > > Lewis Shobbrook > Team Lead - DevOps > > base2Services | The Cloud Services People > T 1300 713 559 E l.shobbr...@base2services.com > Lvl 21, 303 Collins St, Melbourne VIC 3000 > base2services.com.au > > > > > On Mon, 5 Jul 2021 at 18:19, > wrote: > > > > Hello Lew, > > > > How exactly are you testing the tunnel? > > Also, please
Re: [strongSwan] AWS EC2 IKEv2 tunnel up but no throughput
Hello Lew,
How exactly are you testing the tunnel?
Also, please provide the output of iptables-save.
Kind regards
Noel
Am July 5, 2021 7:28:19 AM UTC schrieb Lewis Shobbrook
:
>Hi Guys,
>I have an IKEv2 tunnel that is established and up, but I am unable to
>route any packets across it.
>All ACL's are configured to allow UDP 500,4500 & protocols 50, 51 &
>icmp to/from the non aws end.
>Local iptables are permissive with default policys ACCEPT
>Security groups also allow anything outbound and the above ports &
>protos inbound.
>Here are a few particulars typically requested ahead of time.
>ip_forward is enabled rp_filter disabled as follows...
>net.ipv4.ip_forward = 1
>net.ipv4.conf.all.send_redirects = 0
>net.ipv4.conf.default.send_redirects = 0
>net.ipv4.tcp_max_syn_backlog = 1280
>net.ipv4.icmp_echo_ignore_broadcasts = 1
>net.ipv4.conf.all.accept_source_route = 0
>net.ipv4.conf.all.accept_redirects = 0
>net.ipv4.conf.all.secure_redirects = 0
>net.ipv4.conf.all.log_martians = 1
>net.ipv4.conf.default.accept_source_route = 0
>net.ipv4.conf.default.accept_redirects = 0
>net.ipv4.conf.default.secure_redirects = 0
>net.ipv4.icmp_echo_ignore_broadcasts = 1
>net.ipv4.icmp_ignore_bogus_error_responses = 1
>net.ipv4.tcp_syncookies = 1
>net.ipv4.conf.all.rp_filter = 0
>net.ipv4.conf.default.rp_filter = 0
>net.ipv4.tcp_mtu_probing = 1
>
>tcpdump just shows one way requests
>Looking at the 220 rt table I can see that the auto added route
>appears to be correct, and the xfrm policy nothing obvious to me, with
>no xfrm vi's used.
>Obfuscated ip's naturally...
>ip r li ta 220
>198.168.248.0/29 via 48.138.201.65 dev eth0 proto static src
>48.138.201.70
>ip xfrm policy
>src 48.138.201.64/26 dst 198.168.248.0/29
>dir out priority 371839 ptype main
>tmpl src 48.138.201.70 dst 68.169.15.170
>proto esp spi 0x2c1e849e reqid 1 mode tunnel
>src 198.168.248.0/29 dst 48.138.201.64/26
>dir fwd priority 371839 ptype main
>tmpl src 68.148.15.170 dst 48.138.201.70
>proto esp reqid 1 mode tunnel
>src 198.168.248.0/29 dst 48.138.201.64/26
>dir in priority 371839 ptype main
>tmpl src 68.148.15.170 dst 48.138.201.70
>proto esp reqid 1 mode tunnel
>src 0.0.0.0/0 dst 0.0.0.0/0
>socket in priority 0 ptype main
>src 0.0.0.0/0 dst 0.0.0.0/0
>socket out priority 0 ptype main
>src 0.0.0.0/0 dst 0.0.0.0/0
>socket in priority 0 ptype main
>src 0.0.0.0/0 dst 0.0.0.0/0
>socket out priority 0 ptype main
>src ::/0 dst ::/0
>socket in priority 0 ptype main
>src ::/0 dst ::/0
>socket out priority 0 ptype main
>src ::/0 dst ::/0
>socket in priority 0 ptype main
>src ::/0 dst ::/0
>socket out priority 0 ptype main
>src 198.168.248.0/29 dst 48.138.201.64/26
>dir fwd priority 371840 ptype main
>tmpl src 68.148.15.170 dst 48.138.201.70
>proto esp reqid 2 mode tunnel
>src 198.168.248.0/29 dst 48.138.201.64/26
>dir in priority 371840 ptype main
>tmpl src 68.148.15.170 dst 48.138.201.70
>proto esp reqid 2 mode tunnel
>src 48.138.201.64/26 dst 198.168.248.0/29
>dir out priority 371840 ptype main
>tmpl src 48.138.201.70 dst 68.169.15.170
>proto esp reqid 2 mode tunnel
>
>ipsec statusall
>Status of IKE charon daemon (strongSwan 5.7.2, Linux
>4.14.232-177.418.amzn2.x86_64, x86_64):
> uptime: 54 minutes, since Jul 05 06:10:30 2021
> malloc: sbrk 2846720, mmap 0, used 1023696, free 1823024
> worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0,
>scheduled: 1
> loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4
>md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1
>pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp
>curve25519 chapoly xcbc cmac hmac ctr ccm gcm curl attr kernel-netlink
>resolve socket-default farp stroke vici updown eap-identity eap-sim
>eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2
>eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic
>xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
>Listening IP addresses:
> 48.138.201.70
>Connections:
>tunnel1: %any...68.148.15.170 IKEv2
>tunnel1: local: uses pre-shared key authentication
>tunnel1: remote: uses pre-shared key authentication
>tunnel1: child: 48.138.201.64/26 === 198.168.248.0/29 TUNNEL
>Security Associations (1 up, 0 connecting):
>tunnel1[2]: ESTABLISHED 24 minutes ago,
>48.138.201.70[48.138.201.70]...65.169.15.170[65.169.15.170]
>tunnel1[2]: IKEv2 SPIs: d3e732eb14d78aec_i* b06860d1ceee2f9a_r,
>rekeying disabled
>tunnel1[2]: IKE proposal: AES_GCM_16_256/PRF_HMAC_SHA2_384/ECP_384
>tunnel1{2}: INSTALLED, TUNNEL, reqid 2, ESP in UDP SPIs: cb651f89_i
>479cff91_o
>tunnel1{2}: AES_GCM_16_256, 0 bytes_i, 0 bytes_o, rekeying disabled
>tunnel1{2}: 48.138.201.64/26 === 198.168.248.0/30
>
>VPC flow logs show no proto 50, only 4500 & 500.
>I've also tried to clamp mss not that I expect it would have changed 0
>throughput
>iptables -t mangle -A FORWARD
Re: [strongSwan] AWS EC2 IKEv2 tunnel up but no throughput
Thanks for your reply Noel, Landed in my spam folder... I'm testing with a curl to a known endpoint from the vpn gateway itself and also from the associated local subnet that is accepted on the other side. Here's the output of iptables-save which has changed quite a bit over various efforts to realise throughput. # Generated by iptables-save v1.8.4 on Mon Jul 5 09:52:05 2021 *mangle :PREROUTING ACCEPT [131559:43589022] :INPUT ACCEPT [129402:43459544] :FORWARD ACCEPT [2155:129300] :OUTPUT ACCEPT [131976:29447953] :POSTROUTING ACCEPT [134131:29577253] -A FORWARD -p tcp -m policy --dir in --pol ipsec -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS --set-mss 1360 -A FORWARD -p tcp -m policy --dir out --pol ipsec -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS --set-mss 1360 COMMIT # Completed on Mon Jul 5 09:52:05 2021 # Generated by iptables-save v1.8.4 on Mon Jul 5 09:52:05 2021 *nat :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] -A POSTROUTING -m policy --dir out --pol ipsec -j ACCEPT -A POSTROUTING -s 48.138.201.64/26 -o eth0 -m policy --dir out --pol ipsec -j ACCEPT COMMIT # Completed on Mon Jul 5 09:52:05 2021 # Generated by iptables-save v1.8.4 on Mon Jul 5 09:52:05 2021 *filter :INPUT ACCEPT [42760:14832665] :FORWARD ACCEPT [774:46440] :OUTPUT ACCEPT [43602:9751014] COMMIT Cheers, Lew Lewis Shobbrook Team Lead - DevOps base2Services | The Cloud Services People T 1300 713 559 E l.shobbr...@base2services.com Lvl 21, 303 Collins St, Melbourne VIC 3000 base2services.com.au On Mon, 5 Jul 2021 at 18:19, wrote: > > Hello Lew, > > How exactly are you testing the tunnel? > Also, please provide the output of iptables-save. > > Kind regards > Noel > > Am July 5, 2021 7:28:19 AM UTC schrieb Lewis Shobbrook > : >> >> Hi Guys, >> I have an IKEv2 tunnel that is established and up, but I am unable to >> route any packets across it. >> All ACL's are configured to allow UDP 500,4500 & protocols 50, 51 & >> icmp to/from the non aws end. >> Local iptables are permissive with default policys ACCEPT >> Security groups also allow anything outbound and the above ports & >> protos inbound. >> Here are a few particulars typically requested ahead of time. >> ip_forward is enabled rp_filter disabled as follows... >> net.ipv4.ip_forward = 1 >> net.ipv4.conf.all.send_redirects = 0 >> net.ipv4.conf.default.send_redirects = 0 >> net.ipv4.tcp_max_syn_backlog = 1280 >> net.ipv4.icmp_echo_ignore_broadcasts = 1 >> net.ipv4.conf.all.accept_source_route = 0 >> net.ipv4.conf.all.accept_redirects = 0 >> net.ipv4.conf.all.secure_redirects = 0 >> net.ipv4.conf.all.log_martians = 1 >> net.ipv4.conf.default.accept_source_route = 0 >> net.ipv4.conf.default.accept_redirects = 0 >> net.ipv4.conf.default.secure_redirects = 0 >> net.ipv4.icmp_echo_ignore_broadcasts = 1 >> net.ipv4.icmp_ignore_bogus_error_responses = 1 >> net.ipv4.tcp_syncookies = 1 >> net.ipv4.conf.all.rp_filter = 0 >> net.ipv4.conf.default.rp_filter = 0 >> net.ipv4.tcp_mtu_probing = 1 >> >> tcpdump just shows one way requests >> Looking at the 220 rt table I can see that the auto added route >> appears to be correct, and the xfrm policy nothing obvious to me, with >> no xfrm vi's used. >> Obfuscated ip's naturally... >> ip r li ta 220 >> 198.168.248.0/29 via 48.138.201.65 dev eth0 proto static src 48.138.201.70 >> ip xfrm policy >> src 48.138.201.64/26 dst 198.168.248.0/29 >> dir out priority 371839 ptype main >> tmpl src 48.138.201.70 dst 68.169.15.170 >> proto esp spi 0x2c1e849e reqid 1 mode tunnel >> src 198.168.248.0/29 dst 48.138.201.64/26 >> dir fwd priority 371839 ptype main >> tmpl src 68.148.15.170 dst 48.138.201.70 >> proto esp reqid 1 mode tunnel >> src 198.168.248.0/29 dst 48.138.201.64/26 >> dir in priority 371839 ptype main >> tmpl src 68.148.15.170 dst 48.138.201.70 >> proto esp reqid 1 mode tunnel >> src 0.0.0.0/0 dst 0.0.0.0/0 >> socket in priority 0 ptype main >> src 0.0.0.0/0 dst 0.0.0.0/0 >> socket out priority 0 ptype main >> src 0.0.0.0/0 dst 0.0.0.0/0 >> socket in priority 0 ptype main >> src 0.0.0.0/0 dst 0.0.0.0/0 >> socket out priority 0 ptype main >> src ::/0 dst ::/0 >> socket in priority 0 ptype main >> src ::/0 dst ::/0 >> socket out priority 0 ptype main >> src ::/0 dst ::/0 >> socket in priority 0 ptype main >> src ::/0 dst ::/0 >> socket out priority 0 ptype main >> src 198.168.248.0/29 dst 48.138.201.64/26 >> dir fwd priority 371840 ptype main >> tmpl src 68.148.15.170 dst 48.138.201.70 >> proto esp reqid 2 mode tunnel >> src 198.168.248.0/29 dst 48.138.201.64/26 >> dir in priority 371840 ptype main >> tmpl src 68.148.15.170 dst 48.138.201.70 >> proto esp reqid 2 mode tunnel >> src 48.138.201.64/26 dst 198.168.248.0/29 >> dir out priority 371840 ptype main >> tmpl src 48.138.201.70 dst 68.169.15.170 >> proto esp reqid 2
[strongSwan] AWS EC2 IKEv2 tunnel up but no throughput
Hi Guys,
I have an IKEv2 tunnel that is established and up, but I am unable to
route any packets across it.
All ACL's are configured to allow UDP 500,4500 & protocols 50, 51 &
icmp to/from the non aws end.
Local iptables are permissive with default policys ACCEPT
Security groups also allow anything outbound and the above ports &
protos inbound.
Here are a few particulars typically requested ahead of time.
ip_forward is enabled rp_filter disabled as follows...
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.tcp_max_syn_backlog = 1280
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.tcp_mtu_probing = 1
tcpdump just shows one way requests
Looking at the 220 rt table I can see that the auto added route
appears to be correct, and the xfrm policy nothing obvious to me, with
no xfrm vi's used.
Obfuscated ip's naturally...
ip r li ta 220
198.168.248.0/29 via 48.138.201.65 dev eth0 proto static src 48.138.201.70
ip xfrm policy
src 48.138.201.64/26 dst 198.168.248.0/29
dir out priority 371839 ptype main
tmpl src 48.138.201.70 dst 68.169.15.170
proto esp spi 0x2c1e849e reqid 1 mode tunnel
src 198.168.248.0/29 dst 48.138.201.64/26
dir fwd priority 371839 ptype main
tmpl src 68.148.15.170 dst 48.138.201.70
proto esp reqid 1 mode tunnel
src 198.168.248.0/29 dst 48.138.201.64/26
dir in priority 371839 ptype main
tmpl src 68.148.15.170 dst 48.138.201.70
proto esp reqid 1 mode tunnel
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src ::/0 dst ::/0
socket in priority 0 ptype main
src ::/0 dst ::/0
socket out priority 0 ptype main
src ::/0 dst ::/0
socket in priority 0 ptype main
src ::/0 dst ::/0
socket out priority 0 ptype main
src 198.168.248.0/29 dst 48.138.201.64/26
dir fwd priority 371840 ptype main
tmpl src 68.148.15.170 dst 48.138.201.70
proto esp reqid 2 mode tunnel
src 198.168.248.0/29 dst 48.138.201.64/26
dir in priority 371840 ptype main
tmpl src 68.148.15.170 dst 48.138.201.70
proto esp reqid 2 mode tunnel
src 48.138.201.64/26 dst 198.168.248.0/29
dir out priority 371840 ptype main
tmpl src 48.138.201.70 dst 68.169.15.170
proto esp reqid 2 mode tunnel
ipsec statusall
Status of IKE charon daemon (strongSwan 5.7.2, Linux
4.14.232-177.418.amzn2.x86_64, x86_64):
uptime: 54 minutes, since Jul 05 06:10:30 2021
malloc: sbrk 2846720, mmap 0, used 1023696, free 1823024
worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0,
scheduled: 1
loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4
md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1
pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp
curve25519 chapoly xcbc cmac hmac ctr ccm gcm curl attr kernel-netlink
resolve socket-default farp stroke vici updown eap-identity eap-sim
eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2
eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic
xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
Listening IP addresses:
48.138.201.70
Connections:
tunnel1: %any...68.148.15.170 IKEv2
tunnel1: local: uses pre-shared key authentication
tunnel1: remote: uses pre-shared key authentication
tunnel1: child: 48.138.201.64/26 === 198.168.248.0/29 TUNNEL
Security Associations (1 up, 0 connecting):
tunnel1[2]: ESTABLISHED 24 minutes ago,
48.138.201.70[48.138.201.70]...65.169.15.170[65.169.15.170]
tunnel1[2]: IKEv2 SPIs: d3e732eb14d78aec_i* b06860d1ceee2f9a_r,
rekeying disabled
tunnel1[2]: IKE proposal: AES_GCM_16_256/PRF_HMAC_SHA2_384/ECP_384
tunnel1{2}: INSTALLED, TUNNEL, reqid 2, ESP in UDP SPIs: cb651f89_i 479cff91_o
tunnel1{2}: AES_GCM_16_256, 0 bytes_i, 0 bytes_o, rekeying disabled
tunnel1{2}: 48.138.201.64/26 === 198.168.248.0/30
VPC flow logs show no proto 50, only 4500 & 500.
I've also tried to clamp mss not that I expect it would have changed 0
throughput
iptables -t mangle -A FORWARD -m policy --pol ipsec --dir in -p tcp -m
tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS
--set-mss 1360
I've spent hours searching but have not found anything to help.
Hoping someone here may have a suggestion ot two?
Cheers,
Lew
