Hi Neale, I was able to make a good progress for the GRE-over-IPSec use case. But stumbled at the last step. I have explained the problem faced (an extra IP header added by VPP, overall 4 IP headers in the pkt) and other details about logs and configuration below. Also attached the decrypted pcap (frame 4 shows the problem)
Kindly let me know if you can catch the problem at the VPP side.
*Topology*
*============*
Strongswan <====================================> VPP
loopback 7.7.7.7
loopback 8.8.8.8
GRE 44.44.44.44
GRE src 42.42.42.42 (tunnel remote 44.44.44.44)
IPSEC 20.20.99.215
IPSEC 20.20.99.99
create gre tunnel src 42.42.42.42 instance 1
multipoint
set interface state gre1 up
set interface ip addr gre1 2.2.2.2/32
create teib gre1 peer 2.2.2.1 nh 44.44.44.44
ip route add 7.7.7.7/32 via 2.2.2.1 gre1 (added route
to the overlay 7.7.7.7 of SS)
/* added below commands after IPSec is UP */
set interface state ipip0 up
ip route add 44.44.44.44/32 via ipip0
set interface unnumbered ipip0 use
VirtualFuncEthernet0/7/0.1556
Problem faced
==============
--- SS sends ICMP over GRE over IPSEC. The VPP also replies with
ICMP-o-GRE-o-IPSEC. But the SS was dropping the packets. Debugging the SS
logs did not show any decryption failure. I decrypted the packets captured
at SS. On applying the encrytion and auth keys in the wireshark, it
was showing an extra IP header added to the reply packet making it a total
of 4 IP headers which is wrong
--- I observed that VPP had added GRE tunnel IP header on top of the inner
most packet. Again on top of the GRE IP header, another IP was added by VPP
with SRC and DST being same as that of IPSec tunnel header
(20.20.99.99-to-20.20.99.215). Now on top of this extra IP header
the ESP encapsulation was added with outermost IP header IPs of that of the
IPSec tunnel (20.20.99.99-to-20.20.99.215)
--- Why is VPP adding an extra header on top of GRE IP header. Is this
because of the route adjacencies that has gone bad causing an extra IP
header to be added?
--- Below I have pasted the FIB entry for the destination 7.7.7.7 that are
stacked on FIB entry 73 and 75
--- Attached IPSec decoded pcap, please check frame 4 that is the ICMP
reply from VPP to SS
--- Is there anything wrong in the way I have added routes on the VPP side
for the overlay 7.7.7.7 and GRE destination 44.44.44.44
Ping on SS
========
root@SS# ping 8.8.8.8 -I 7.7.7.7 -c 3
PING 8.8.8.8 (8.8.8.8) from 7.7.7.7 : 56(84) bytes of data.
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2041ms
TCPDUMP taken on SS
=======================
12:46:48.439122 IP vmsrvrlnx-strongswan-215 > dns.google: ICMP echo
request, id 43, seq 1, length 64
12:46:48.439174 IP vmsrvrlnx-strongswan-215 > 20.20.99.99:
ESP(spi=0x1234567b,seq=0x1), length 148
12:46:48.439178 ethertype IPv4, IP vmsrvrlnx-strongswan-215 > 20.20.99.99:
ESP(spi=0x1234567b,seq=0x1), length 148
12:46:48.439487 ethertype IPv4, IP 20.20.99.99 > vmsrvrlnx-strongswan-215:
ESP(spi=0xce90b744,seq=0x1), length 180
12:46:48.439487 IP 20.20.99.99 > vmsrvrlnx-strongswan-215:
ESP(spi=0xce90b744,seq=0x1), length 180
12:46:49.455775 IP vmsrvrlnx-strongswan-215 > dns.google: ICMP echo
request, id 43, seq 2, length 64
12:46:49.455826 IP vmsrvrlnx-strongswan-215 > 20.20.99.99:
ESP(spi=0x1234567b,seq=0x2), length 148
12:46:49.455831 ethertype IPv4, IP vmsrvrlnx-strongswan-215 > 20.20.99.99:
ESP(spi=0x1234567b,seq=0x2), length 148
12:46:49.455958 ethertype IPv4, IP 20.20.99.99 > vmsrvrlnx-strongswan-215:
ESP(spi=0xce90b744,seq=0x2), length 180
12:46:49.455958 IP 20.20.99.99 > vmsrvrlnx-strongswan-215:
ESP(spi=0xce90b744,seq=0x2), length 180
12:46:50.479872 IP vmsrvrlnx-strongswan-215 > dns.google: ICMP echo
request, id 43, seq 3, length 64
12:46:50.479933 IP vmsrvrlnx-strongswan-215 > 20.20.99.99:
ESP(spi=0x1234567b,seq=0x3), length 148
12:46:50.479939 ethertype IPv4, IP vmsrvrlnx-strongswan-215 > 20.20.99.99:
ESP(spi=0x1234567b,seq=0x3), length 148
12:46:50.480080 ethertype IPv4, IP 20.20.99.99 > vmsrvrlnx-strongswan-215:
ESP(spi=0xce90b744,seq=0x3), length 180
12:46:50.480080 IP 20.20.99.99 > vmsrvrlnx-strongswan-215:
ESP(spi=0xce90b744,seq=0x3), length 180
VPP LOGS
=================================
vpp# show gre tunnel
[0] instance 1 src 42.42.42.42 dst 0.0.0.0 fib-idx 0 sw-if-idx 20 payload
L3 multi-point
vpp# show teib
[0] gre1:2.2.2.1 via [0]:44.44.44.44/32
vpp# show ipip tunnel
[0] instance 0 src 20.20.99.99 dst 20.20.99.215 table-ID 0 sw-if-idx 21
flags [none] dscp CS0
FIB ouput of loopback IPs and GRE tunnel IPs shown in topology
=============================================
7.7.7.7/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:75 buckets:1 uRPF:105 to:[0:0]]
[0] [@6]: ipv4 via 2.2.2.1 gre1: mtu:9000 next:15
4500000000000000fe2f10232a2a2a2a2c2c2c2c00000800
stacked-on entry:73:
[@3]: ipv4 via 0.0.0.0 ipip0: mtu:9000 next:17
450000000000000040048b9814146363141463d7
stacked-on entry:75:
[@2]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556:
mtu:9000 next:16 fa163e4b6b42fa163ec2b4f4810006140800
8.8.8.8/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:72 buckets:1 uRPF:103 to:[0:0]]
[0] [@2]: dpo-receive: 8.8.8.8 on loop3
42.42.42.42/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:71 buckets:1 uRPF:102 to:[0:0]]
[0] [@2]: dpo-receive: 42.42.42.42 on loop2
44.44.44.44/32
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:74 buckets:1 uRPF:107 to:[0:0]]
[0] [@6]: ipv4 via 0.0.0.0 ipip0: mtu:9000 next:17
450000000000000040048b9814146363141463d7
stacked-on entry:75:
[@2]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556:
mtu:9000 next:16 fa163e4b6b42fa163ec2b4f4810006140800
vpp#
vpp# show fib entry 73
73@44.44.44.44/32 fib:0 index:73 locks:4
CLI refs:1 entry-flags:attached, src-flags:added,contributing,active,
path-list:[81] locks:2 flags:shared, uPRF-list:107 len:1 itfs:[21, ]
path:[117] pl-index:81 ip4 weight=1 pref=0 attached-nexthop:
oper-flags:resolved, cfg-flags:attached,
44.44.44.44 ipip0 (p2p)
[@0]: ipv4 via 0.0.0.0 ipip0: mtu:9000 next:17
450000000000000040048b9814146363141463d7
stacked-on entry:75:
[@2]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556:
mtu:9000 next:16 fa163e4b6b42fa163ec2b4f4810006140800
recursive-resolution refs:1 src-flags:added, cover:-1
forwarding: unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:74 buckets:1 uRPF:107 to:[0:0]]
[0] [@6]: ipv4 via 0.0.0.0 ipip0: mtu:9000 next:17
450000000000000040048b9814146363141463d7
stacked-on entry:75:
[@2]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556:
mtu:9000 next:16 fa163e4b6b42fa163ec2b4f4810006140800
Delegates:
track: sibling:147
Children:{adj:17}
Children:{fib-entry-track:16}
vpp#
vpp#
vpp#
vpp# show fib entry 75
75@20.20.99.215/32 fib:0 index:75 locks:4
adjacency refs:1 entry-flags:attached,
src-flags:added,contributing,active, cover:65
path-list:[80] locks:2 uPRF-list:106 len:1 itfs:[17, ]
path:[116] pl-index:80 ip4 weight=1 pref=0 attached-nexthop:
oper-flags:resolved,
20.20.99.215 VirtualFuncEthernet0/7/0.1556
[@0]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556: mtu:9000
next:16 fa163e4b6b42fa163ec2b4f4810006140800
Extensions:
path:116 adj-flags:[refines-cover]
recursive-resolution refs:1 src-flags:added, cover:-1
forwarding: unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:76 buckets:1 uRPF:106
to:[14:3252]]
[0] [@5]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556: mtu:9000
next:16 fa163e4b6b42fa163ec2b4f4810006140800
Delegates:
track: sibling:157
Children:{adj:19}{ipsec-sa:0}
Children:{fib-entry-track:17}
vpp#
vpp#
Regards,
Vijay
On Tue, Mar 23, 2021 at 5:44 PM Neale Ranns <ne...@graphiant.com> wrote:
>
>
> Hi Vijay,
>
>
>
> I don’t think you can have both a GRE and IPIP tunnel to the same peer,
> sourced from the same local address. These two tunnels will create two
> identical ‘keys’ in the lookup table (src:X,dst:Y,proto:ESP). my guess is
> your packet matches against the IPIP tunnel, not the GRE and the key
> material is different, hence the integ failure. You can see in the trace
> that it used SA:1, check ‘sh ipsec sa 1’ and ‘sh ipsec protect’ to see
> which tunnel that is associated with. You can also see the lookup table
> with ‘sh ipsec protect-hash’.
>
>
>
> If you remove (or admin down) the ipip tunnel, does it work?
>
>
>
> /neale
>
>
>
>
>
> *From: *Vijay Kumar <vjkumar2...@gmail.com>
> *Date: *Tuesday, 23 March 2021 at 04:18
> *To: *Neale Ranns <ne...@graphiant.com>
> *Cc: *vpp-dev <vpp-dev@lists.fd.io>
> *Subject: *Re: [vpp-dev] GRE-over-IPSec fails
>
> Hi Neale,
>
>
>
> Could you let me know if you faced the mentioned problem anytime?
>
>
>
> For me only IPSec works fine, Only GRE also works fine. But when I
> configure GRE-over-IPSec, the traffic is dropped at *esp4-decrypt-tun*
> due to integrity check failure.
>
> As there are two logical interfaces created at VPP (ipip0 and gre0) for
> the peer, do I need to take care of something specially? As far as I know,
> I haven't missed any config.
>
>
>
>
>
> Regards,
>
> Vijay Kumar N
>
>
>
> On Mon, Mar 22, 2021 at 11:31 PM Vijay Kumar via lists.fd.io <vjkumar2003=
> gmail....@lists.fd.io> wrote:
>
> Hi,
>
>
>
> I am trying a test case where-in I have an GRE P2MP (mGRE) tunnel on the
> VPP. The GRE peer is a strongswan VM that hosts both the GRE tunnel and
> IPSec SA. When I started ping traffic from SS, the traffic is dropped at
> esp4-decrypt-tun graph node due to integrity check failure.
>
>
>
> Has any one tested GRE-over-IPSec recently? If so can you pls share me a
> working config. If not please review the below config and let me know if I
> missed something
>
>
>
> *NOTE: -*
>
> If I have run only GRE test case, traffic is fine (no IPSec enabled). If I
> have only IPSec configured but no GRE then also traffic is fine.
>
>
>
> I am facing this issue only when both GRE and IPSec are enabled at the
> same time.
>
>
>
> Topology and config at SS and VPP
>
> ==============================
>
> Strongswan VM (20.20.99.215, gre peer 2.2.2.1, loopback 7.7.7.7)
> <=============> VPP cluster (20.20.99.99, gre peer 2.2.2.2, loopback
> 8.8.8.8)
>
> IPSec SA Traffic Selector (7.7.7.7/32 to 8.8.8.8/32)
>
> ike=aes256-sha256-modp2048!
>
> esp=aes256-sha1-noesn!
>
>
>
>
>
> Below is the VPP trace
>
> ================
>
> 03:20:34:670201: dpdk-input
> VirtualFuncEthernet0/7/0 rx queue 0
> buffer 0x4c6b91: current data 0, length 170, buffer-pool 0, ref-count 1,
> totlen-nifb 0, trace handle 0x1000000
> ext-hdr-valid
> l4-cksum-computed l4-cksum-correct
> PKT MBUF: port 0, nb_segs 1, pkt_len 170
> buf_len 2176, data_len 170, ol_flags 0x180, data_off 128, phys_addr
> 0xa3dae4c0
> packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
> rss 0x0 fdir.hi 0x0 fdir.lo 0x0
> Packet Offload Flags
> PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
> PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
> Packet Types
> RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
> RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without
> extension headers
> RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet
> IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:c2:b4:f4 802.1q vlan 1556
> IPSEC_ESP: 20.20.99.215 -> 20.20.99.99
> tos 0x00, ttl 64, length 152, checksum 0x5b33 dscp CS0 ecn NON_ECN
> fragment id 0xef9e, flags DONT_FRAGMENT
> 03:20:34:670208: ethernet-input
> frame: flags 0x3, hw-if-index 3, sw-if-index 3
> IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:c2:b4:f4 802.1q vlan 1556
> 03:20:34:670214: ip4-input
> IPSEC_ESP: 20.20.99.215 -> 20.20.99.99
> tos 0x00, ttl 64, length 152, checksum 0x5b33 dscp CS0 ecn NON_ECN
> fragment id 0xef9e, flags DONT_FRAGMENT
> 03:20:34:670218: ip4-lookup
> fib 1 dpo-idx 21 flow hash: 0x00000000
> IPSEC_ESP: 20.20.99.215 -> 20.20.99.99
> tos 0x00, ttl 64, length 152, checksum 0x5b33 dscp CS0 ecn NON_ECN
> fragment id 0xef9e, flags DONT_FRAGMENT
> 03:20:34:670220: ip4-local
> IPSEC_ESP: 20.20.99.215 -> 20.20.99.99
> tos 0x00, ttl 64, length 152, checksum 0x5b33 dscp CS0 ecn NON_ECN
> fragment id 0xef9e, flags DONT_FRAGMENT
> 03:20:34:670222: ipsec4-tun-input
> IPSec: remote:20.20.99.215 spi:305419897 (0x12345679) seq 40 sa 1
> 03:20:34:670225: esp4-decrypt-tun
> esp: crypto aes-cbc-256 integrity sha1-96 pkt-seq 40 sa-seq 0 sa-seq-hi 0
> 03:20:34:670241: ip4-drop
> IP6_NONXT: 242.163.36.86 -> 70.168.225.19
> version 1, header length 8
> tos 0x34, ttl 245, length 22137, checksum 0x5156 (should be 0x972a)
> dscp unknown ecn NON_ECN
> fragment id 0x0000 offset 320
> 03:20:34:670243: error-drop
> rx:ipip0
> 03:20:34:670244: drop
> esp4-decrypt-tun: Integrity check failed
>
>
>
> vpp# show node counters
> Count Node Reason
> 25 esp4-encrypt-tun ESP pkts received
> 213 memif-input not ip packet
> 3 dpdk-input no error
> 136 arp-reply ARP replies sent
> 3 arp-reply IP4 source address not
> local to subnet
> 1 gre4-input no error
> 213 ip4-udp-lookup No error
> 42 esp4-decrypt-tun ESP pkts received
> 42 esp4-decrypt-tun Integrity check failed
> 25 esp4-encrypt-tun ESP pkts received
> 42 ipsec4-tun-input good packets received
> 11 ip4-local ip4 source lookup miss
> 3 ip4-local unknown ip protocol
> 3 ethernet-input unknown vlan
> vpp#
>
>
>
>
>
>
>
>
>
>
>
VPP adding extra IP header in the GRE-over-IPSec.pcap
Description: Binary data
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19019): https://lists.fd.io/g/vpp-dev/message/19019 Mute This Topic: https://lists.fd.io/mt/81531694/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
