Hi Ole,
Thanks for looking into this.
I'm seeing repeated NAT64 and ACL-related errors, plus intermittent DPDK TX
failures on TwentyFiveGigabitEthernet81/0/0. Full output:
Count Node Reason
Severity
7 acl-plugin-out-ip6-fa ACL permit packets error
7 acl-plugin-out-ip6-fa checked packets error
3 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error
1 VirtualEthernet0/0/3-output interface is down error
7621 nat64-out2in-handoff same worker
error
5179 nat64-out2in-handoff do handoff
error
6745 nat64-out2in no translation
error
13 nat64-in2out-handoff do handoff error
65271 dpdk-input no error
error
14 acl-plugin-out-ip6-fa ACL deny packets error
2950 acl-plugin-out-ip6-fa ACL permit packets
error
2964 acl-plugin-out-ip6-fa checked packets
error
73336 arp-reply ARP replies sent
info
2 arp-reply ARP request IP4 source address lear info
10 ip6-glean neighbor solicitations sent info
2 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error
15 ip6-icmp-input neighbor solicitations for unknown error
9 ip6-icmp-input neighbor advertisements sent info
2 ip6-icmp-input neighbor advertisements received info
920 nat64-out2in no translation
error
2565 nat64-in2out-handoff do handoff
error
24 acl-plugin-out-ip6-fa ACL permit packets error
24 acl-plugin-out-ip6-fa checked packets error
4 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error
2 TwentyFiveGigabitEthernet81/0/0-tx Tx packet drops (dpdk tx failure) error
765 nat64-out2in no translation
error
2 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error
461 nat64-out2in no translation
error
2 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error
331 TwentyFiveGigabitEthernet81/0/0-tx Tx packet drops (dpdk tx failure)
error
624 nat64-out2in no translation
error
981 nat64-in2out-handoff do handoff
error
2 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error
531 nat64-out2in no translation
error
12 nat64-in2out-handoff do handoff error
2 ip6-local-hop-by-hop Unknown protocol ip6 local h-b-h pa error
Here's my sessions + bib table
------------------------------
NAT64 sessions:
2604:2dc0:400:dddd:0:2:0:20 51568 64:ff9b::369d:dd0 443 15.220.168.42 5832
54.157.13.208 443 protcol tcp vrf 3
2604:2dc0:400:dddd:0:2:0:20 51564 64:ff9b::369d:dd0 443 15.220.168.42 6903
54.157.13.208 443 protcol tcp vrf 3
2604:2dc0:400:dddd:0:2:0:20 51562 64:ff9b::369d:dd0 443 15.220.168.42 5757
54.157.13.208 443 protcol tcp vrf 3
2604:2dc0:400:dddd:0:2:0:22 42380 64:ff9b::8b3c:a058 5205 15.220.168.42 42400
139.60.160.88 5205 protcol tcp vrf 3
2604:2dc0:400:dddd:0:2:0:22 42366 64:ff9b::8b3c:a058 5205 15.220.168.42 39831
139.60.160.88 5205 protcol tcp vrf 3
NAT64 BIB entries:
2604:2dc0:400:dddd:0:2:0:20 51568 15.220.168.42 5832 protocol tcp vrf 3 dynamic
1 sessions
2604:2dc0:400:dddd:0:2:0:20 51564 15.220.168.42 6903 protocol tcp vrf 3 dynamic
1 sessions
2604:2dc0:400:dddd:0:2:0:20 51562 15.220.168.42 5757 protocol tcp vrf 3 dynamic
1 sessions
2604:2dc0:400:dddd:0:2:0:22 42380 15.220.168.42 42400 protocol tcp vrf 3
dynamic 1 sessions
2604:2dc0:400:dddd:0:2:0:22 42366 15.220.168.42 39831 protocol tcp vrf 3
dynamic 1 sessions
I tried to capture an E2E flow, while I think both traces are not from the same
request, so the delay between them might not properly transcribe the handoff
duration.
17:41:50:085312: vhost-user-input
VirtualEthernet0/0/2 queue 0
virtio flags:
SINGLE_DESC Single descriptor packet
virtio_net_hdr first_desc_len 130
flags 0x00 gso_type 0
num_buff 0
17:41:50:085316: ethernet-input
frame: flags 0x1, hw-if-index 5, sw-if-index 5
IP6: 02:00:00:da:77:36 -> 02:00:00:da:77:36
17:41:50:085319: ip6-input
ICMP6: 2604:2dc0:400:dddd:0:2:0:22 -> 64:ff9b::808:808
tos 0x00, flow label 0xa0802, hop limit 64, payload length 64
ICMP echo_request checksum 0x3812
17:41:50:085321: ip6-lookup
fib 2 dpo-idx 10 flow hash: 0x00000000
ICMP6: 2604:2dc0:400:dddd:0:2:0:22 -> 64:ff9b::808:808
tos 0x00, flow label 0xa0802, hop limit 64, payload length 64
ICMP echo_request checksum 0x3812
17:41:50:085323: ip6-rewrite
tx_sw_if_index 2 adj-idx 10 : ipv6 via 2001:db8::3:0:1 loop0: mtu:9000 next:4
flags:[] 02640000000302640000000386dd flow hash: 0x00000000
00000000: 02640000000302640000000386dd600a080200403a3f26042dc00400c0010000
00000020: 0002000000220064ff9b00000000000000000808080880003812196f0014275d
00000040: 306900000000e65c090000000000101112131415161718191a1b1c1d1e1f2021
00000060: 22232425262728292a2b2c2d2e2f3031323334353637559fd2fd1a9d
17:41:50:085324: loop0-output
loop0 flags 0x00180005
IP6: 02:64:00:00:00:03 -> 02:64:00:00:00:03
ICMP6: 2604:2dc0:400:dddd:0:2:0:22 -> 64:ff9b::808:808
tos 0x00, flow label 0xa0802, hop limit 63, payload length 64
ICMP echo_request checksum 0x3812
17:41:50:085327: ethernet-input
IP6: 02:64:00:00:00:03 -> 02:64:00:00:00:03
17:41:50:085327: ip6-input
ICMP6: 2604:2dc0:400:dddd:0:2:0:22 -> 64:ff9b::808:808
tos 0x00, flow label 0xa0802, hop limit 63, payload length 64
ICMP echo_request checksum 0x3812
17:41:50:085328: ip6-sv-reassembly-feature
[not fragmented or atomic fragment]
17:41:50:085329: nat64-in2out-handoff
NAT64-IN2OUT-HANDOFF: next-worker 4
17:41:59:104234: handoff_trace
HANDED-OFF: from thread 2 trace index 22
17:41:59:104234: nat64-in2out
NAT64-in2out: sw_if_index 2, next index 0
17:41:59:114299: ip4-lookup
fib 0 dpo-idx 7 flow hash: 0x00000000
ICMP: 15.220.168.42 -> 8.8.8.8
tos 0x00, ttl 63, length 84, checksum 0xb3a3 dscp CS0 ecn NON_ECN
fragment id 0x0000
ICMP echo_request checksum 0x77a0 id 33816
17:41:59:114301: ip4-rewrite
tx_sw_if_index 1 dpo-idx 7 : ipv4 via 15.220.168.46
TwentyFiveGigabitEthernet81/0/0: mtu:9000 next:3 flags:[]
001c7317c0235c2573ac596c0800 flow hash: 0x00000000
00000000: 001c7317c0235c2573ac596c080045000054000000003e01b4a30fcca82a0808
00000020: 0808080077a08418001d305d306900000000d3900900000000001011
17:41:59:114303: TwentyFiveGigabitEthernet81/0/0-output
TwentyFiveGigabitEthernet81/0/0 flags 0x00180005
IP4: 5c:25:73:ac:59:6c -> 00:1c:73:17:c0:23
ICMP: 15.220.168.42 -> 8.8.8.8
tos 0x00, ttl 62, length 84, checksum 0xb4a3 dscp CS0 ecn NON_ECN
fragment id 0x0000
ICMP echo_request checksum 0x77a0 id 33816
17:41:59:114304: TwentyFiveGigabitEthernet81/0/0-tx
TwentyFiveGigabitEthernet81/0/0 tx queue 4
buffer 0xfda0f9: current data 20, length 98, buffer-pool 0, ref-count 1, trace
handle 0x4000009
l2-hdr-offset 0 l3-hdr-offset 14
PKT MBUF: port 65535, nb_segs 1, pkt_len 98
buf_len 2176, data_len 98, ol_flags 0x0, data_off 148, phys_addr 0xbf683ec0
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 5c:25:73:ac:59:6c -> 00:1c:73:17:c0:23
ICMP: 15.220.168.42 -> 8.8.8.8
tos 0x00, ttl 62, length 84, checksum 0xb4a3 dscp CS0 ecn NON_ECN
fragment id 0x0000
ICMP echo_request checksum 0x77a0 id 33816
Ping output shows high RTT:
64 bytes from 64:ff9b::808:808: icmp_seq=184 ttl=113 time=31.1 ms
64 bytes from 64:ff9b::808:808: icmp_seq=185 ttl=113 time=40.1 ms
64 bytes from 64:ff9b::808:808: icmp_seq=186 ttl=113 time=39.1 ms
64 bytes from 64:ff9b::808:808: icmp_seq=187 ttl=113 time=38.1 ms
64 bytes from 64:ff9b::808:808: icmp_seq=188 ttl=113 time=37.0 ms
64 bytes from 64:ff9b::808:808: icmp_seq=189 ttl=113 time=36.1 ms
64 bytes from 64:ff9b::808:808: icmp_seq=190 ttl=113 time=35.1 ms
64 bytes from 64:ff9b::808:808: icmp_seq=191 ttl=113 time=34.1 ms
64 bytes from 64:ff9b::808:808: icmp_seq=192 ttl=113 time=33.1 ms
64 bytes from 64:ff9b::808:808: icmp_seq=193 ttl=113 time=32.1 ms
Mathis
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#26610): https://lists.fd.io/g/vpp-dev/message/26610
Mute This Topic: https://lists.fd.io/mt/116590423/21656
Mute #ipv6:https://lists.fd.io/g/vpp-dev/mutehashtag/ipv6
Mute #nat:https://lists.fd.io/g/vpp-dev/mutehashtag/nat
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
