Hi florin, Thanks a lot for helping me out. I will try your patch and update you with the result.
A general observation ====================== In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD counter is incremented. I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph node tcp46_listen_inline() of VPP 20.05 the SYN pkt trace function is called towards the end i.e. after calling tcp_send_synack() but in 21.06 code, the tcp46_listen_trace_frame() is called at the very beginning of tcp46_listen_inline() graph node. I think moving the pkt trace macro to the end of the function is good (placing it close to the line that calls tcp_send_synack), otherwise it can mislead us and will not help debugging. On Wed, Mar 16, 2022 at 10:22 AM Florin Coras <fcoras.li...@gmail.com> wrote: > Hi Vijay, > > That’s probably because packets are hitting an actual error and it seems > the listen node is not reporting anything but syns received. Here’s a patch > that might help [1]. It might not cherry-pick cleanly on 21.06. > > Regards, > Florin > > [1] https://gerrit.fd.io/r/c/vpp/+/35654 > > On Mar 15, 2022, at 7:54 PM, Vijay Kumar <vjkumar2...@gmail.com> wrote: > > Hi Florin, > > I have the output of show node counters (show errors) taken on both 20.05 > and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or > tcp4-output etc. > > Please let me know why SYN rcvd counter itself is not incremented but in > the earlier reply I already pasted the show trace ouput where we saw the > SYN pkt was landed on tcp4-litsen node. > > VPP 20.05 > =========== > pp# show node counters > Count Node Reason > 3 memif-input not ip packet > 10259 an_ppe_wfectrl wfectrl packets received > 10259 an_ppe_wfectrl wfectrl replies sent > 1 an_ppe_wfectrl wfectrl packet > processing failed > 8123 an_ppe_wfectrl session stat request > received > 1 an_ppe_wfectrl service construct config > request received > 1 an_ppe_wfectrl service construct config > request success > 1 an_ppe_wfectrl service config request > received > 1 an_ppe_wfectrl service config request > success > 1686 an_ppe_wfectrl dpi stats request > received > 1686 an_ppe_wfectrl dpi stats request success > 70 an_ppe_wfectrl nat stats request > received > 70 an_ppe_wfectrl nat stats request success > 70 an_ppe_wfectrl vpp stats request > received > 70 an_ppe_wfectrl vpp stats request success > 1 an_ppe_wfectrl udp tunnel resource > block add request received > 1 an_ppe_wfectrl udp tunnel resource > block add request success > 1 an_ppe_wfectrl l3rm resource block > update request received > 1 an_ppe_wfectrl l3rm resource block > update request success > 1 an_ppe_wfectrl ue registration request > received > 17 an_ppe_wfectrl ike msg received from > ikemgr > 17 an_ppe_wfectrl ike msg send to network > success > 3 an_ppe_wfectrl ipsec sa install msg > received from ikemgr > 3 an_ppe_wfectrl ipsec sa install msg > processed successfully > 1 an_ppe_vppctrl_input Success in insert in > ueidentifier > 1 an_ppe_vppctrl_input Success in insert in > uecontext > 2 an_ppe_vppctrl_input Downlink message sent to > UE successfully > 1 an_ppe_vppctrl_input TCP ESTABLISHMENT > message received > 1 an_ppe_vppctrl_input UPLINK NAS message > received > 1 an_ppe_vppctrl_input NAS TCP CLOSE message > received > 7 an_ppe_router_input packets received in rtr > 7 an_ppe_router_input packets dropped to host. > no tacptcp session found > 17 an-ppe-isakmp4-output Total IKEV4 packets > dispatched to network > 17 an-ppe-isakmpmgr-input packets processed by > isakmpmgr input plugin > 17 an-ppe-isakmpmgr-input Received IKE packets > 17 an-ppe-isakmpmgr-input Successfully sent ike > message to Session Manager > 5 an-ppe-isakmpmgr-input Received ike exchange > AUTH packet > 4 an-ppe-isakmpmgr-input Received ike exchange > CREATE CHILD SA packet > 1 an-ppe-isakmpmgr-input Received ike exchange SA > INIT packet > 7 an-ppe-isakmpmgr-input Received ike exchange > INFORMATIONAL packet > 5 arp-reply ARP replies sent > 4 session-queue Packets transmitted > 17 ip4-udp-lookup No error > 1 tcp4-listen SYNs received > 2 tcp4-rcv-process Pure ACKs received > 1 tcp4-established Packets pushed into rx > fifo > 2 tcp4-established Pure ACKs received > 1 tcp4-established FINs received > 5 tcp4-output Packets sent > 7 esp4-decrypt-tun ESP pkts received > 5 esp4-encrypt-tun ESP pkts received > 5 esp4-encrypt-tun total control pkts > received > 7 ipsec4-tun-input good packets received > 2 ip4-local unknown ip protocol > 5 ethernet-input unknown vlan > vpp# > vpp# > > > > > > VPP 21.06 output > =============== > vpp# > vpp# show version > vpp v21.06.0-3~g2a4af2b5d-dirty built by an-vijay_kumar on 500aaad2d090 at > 2022-03-15T01:16:27 > vpp# > vpp# > vpp# show node counters > Count Node > Reason Severity > 1 null-node blackholed > packets error > 6690 an_ppe_wfectrl wfectrl > packets received error > 6690 an_ppe_wfectrl wfectrl > replies sent error > 1 an_ppe_wfectrl wfectrl packet > processing failed error > 5329 an_ppe_wfectrl session stat > request received error > 1 an_ppe_wfectrl service construct > config request received error > 1 an_ppe_wfectrl service construct > config request success error > 1 an_ppe_wfectrl service config > request received error > 1 an_ppe_wfectrl service config > request success error > 1068 an_ppe_wfectrl dpi stats > request received error > 1068 an_ppe_wfectrl dpi stats > request success error > 44 an_ppe_wfectrl nat stats > request received error > 44 an_ppe_wfectrl nat stats > request success error > 44 an_ppe_wfectrl vpp stats > request received error > 44 an_ppe_wfectrl vpp stats > request success error > 1 an_ppe_wfectrl udp tunnel resource > block add request received error > 1 an_ppe_wfectrl udp tunnel resource > block add request success error > 1 an_ppe_wfectrl l3rm resource block > update request received error > 1 an_ppe_wfectrl l3rm resource block > update request success error > 1 an_ppe_wfectrl ue registration > request received error > 1 an_ppe_wfectrl ue deregistration > request received error > 14 an_ppe_wfectrl ike msg > received from ikemgr error > 14 an_ppe_wfectrl ike msg send to > network success error > 3 an_ppe_wfectrl ipsec sa install msg > received from ikemgr error > 3 an_ppe_wfectrl ipsec sa delete msg > received from ikemgr error > 3 an_ppe_wfectrl ipsec sa install msg > processed successfully error > 3 an_ppe_wfectrl ipsec sa delete msg > processed successfully error > 1 an_ppe_vppctrl_input Success in delete > in ueidentifier error > 1 an_ppe_vppctrl_input Success in insert > in ueidentifier error > 1 an_ppe_vppctrl_input Success in > insert in uecontext error > 1 an_ppe_vppctrl_input Success in > delete in uecontext error > 5 an_ppe_router_input packets > received in rtr error > 5 an_ppe_router_input packets dropped to host. > no tacptcp session found error > 14 an-ppe-isakmp4-output Total IKEV4 packets > dispatched to network error > 14 an-ppe-isakmpmgr-input packets processed by > isakmpmgr input plugin error > 14 an-ppe-isakmpmgr-input Received > IKE packets error > 14 an-ppe-isakmpmgr-input Successfully sent ike > message to Session Manager error > 5 an-ppe-isakmpmgr-input Received ike > exchange AUTH packet error > 4 an-ppe-isakmpmgr-input Received ike exchange > CREATE CHILD SA packet error > 1 an-ppe-isakmpmgr-input Received ike > exchange SA INIT packet error > 4 an-ppe-isakmpmgr-input Received ike exchange > INFORMATIONAL packet error > 3 arp-reply ARP > replies sent error > 14 ip4-udp-lookup No > error error > 5 esp4-decrypt-tun ESP pkts > received error > 5 ipsec4-tun-input good packets > received error > 1 ip4-input ip4 ttl > <= 1 error > 1 ip4-icmp-error hop limit > exceeded response sent error > 27628 ethernet-input unknown > vlan error > vpp# > vpp# > > > > > On Wed, Mar 16, 2022 at 6:52 AM Florin Coras <fcoras.li...@gmail.com> > wrote: > >> Hi Vijay, >> >> You won’t see the syn/syn-acks in traces because of the way they are >> generated. Nonetheless, you can verify that the syns were received with >> “show error” and check that the syn-acks were actually generated with a >> pcap trace rxtx <output interface> >> >> Previously tracing worked because buffers were re-used and incoming >> packets were directly sent to output. More recently we’ve started >> dispatching syn-acks through the session layer in order to minimize size of >> tx bursts per dispatch. >> >> Regards, >> Florin >> >> On Mar 15, 2022, at 6:08 PM, Vijay Kumar <vjkumar2...@gmail.com> wrote: >> >> Hi Florin, >> >> Thanks for the quick response. >> >> It means it is expected to see the SYN pkts getting dropped after it is >> terminated but I was not able to see any SYN-ACK going out of VPP. >> >> In both, "show trace" cmd output and also the pcap taken at dspatch >> graph node level, I was not able to see the SYN-ACK going out. The route >> to the SYN-ACK destination (which is the original source that started SYN) >> is also present in the ip fib output. The configuration is the same that >> was working fine for me in vp 20.05. >> >> Is there anything that I can look at for the SYN-ACK not sending issue? >> >> >> >> On Wed, 16 Mar 2022, 00:40 Florin Coras, <fcoras.li...@gmail.com> wrote: >> >>> Hi Vijay, >>> >>> I see an_ppe_router_input forwards syns to tcp-input and those packets >>> are delivered to tcp-listen, i.e., you’ve created a listener that’s matched >>> by the incoming traffic. >>> >>> The thing to keep in mind is that tcp terminates incoming flows and it >>> does not reuse buffers. That is, the syn hits the listen node and a syn-ack >>> is programmed for this new connection that still needs to complete the >>> handshake. The original syn packet is discarded and therefore you see it as >>> a drop. >>> >>> Regards, >>> Florin >>> >>> On Mar 15, 2022, at 3:50 AM, Vijay Kumar <vjkumar2...@gmail.com> wrote: >>> >>> The is the output of show trace and show interface >>> ============================================ >>> >>> Packet 36 >>> >>> 00:03:26:875694: dpdk-input >>> VirtualFuncEthernet0/7/0 rx queue 0 >>> buffer 0x4c1b21: current data 0, length 138, buffer-pool 0, ref-count >>> 1, totlen-nifb 0, trace handle 0x1000023 >>> ext-hdr-valid >>> l4-cksum-computed l4-cksum-correct >>> PKT MBUF: port 0, nb_segs 1, pkt_len 138 >>> buf_len 2176, data_len 138, ol_flags 0x182, data_off 128, phys_addr >>> 0xc266c8c0 >>> packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 >>> rss 0xde0c21da fdir.hi 0x0 fdir.lo 0xde0c21da >>> Packet Offload Flags >>> PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result >>> PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid >>> PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid >>> Packet Types >>> RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet >>> RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without >>> extension headers >>> RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet >>> IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556 >>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>> fragment id 0x0000, flags DONT_FRAGMENT >>> 00:03:26:875702: ethernet-input >>> frame: flags 0x3, hw-if-index 3, sw-if-index 3 >>> IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556 >>> 00:03:26:875708: ip4-input >>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>> fragment id 0x0000, flags DONT_FRAGMENT >>> 00:03:26:875714: ip4-lookup >>> fib 2 dpo-idx 27 flow hash: 0x00000000 >>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>> fragment id 0x0000, flags DONT_FRAGMENT >>> 00:03:26:875715: ip4-local >>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>> fragment id 0x0000, flags DONT_FRAGMENT >>> 00:03:26:875717: ipsec4-tun-input >>> IPSec: remote:20.20.99.215 spi:4213 (0x00001075) sa:1 tun:0 seq 1 sa >>> -2029505104 >>> 00:03:26:875719: esp4-decrypt-tun >>> esp: crypto aes-cbc-256 integrity sha1-96 pkt-seq 1 sa-seq 1 sa-seq-hi >>> 0 >>> 00:03:26:875801: ip4-input-no-checksum >>> TCP: 44.44.44.44 -> 30.30.30.30 >>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN >>> fragment id 0x692e, flags DONT_FRAGMENT >>> TCP: 45723 -> 1234 >>> seq. 0x119d35d0 ack 0x00000000 >>> flags 0x02 SYN, tcp header: 40 bytes >>> window 64240, checksum 0xd6be >>> 00:03:26:875803: ip4-lookup >>> fib 2 dpo-idx 25 flow hash: 0x00000000 >>> TCP: 44.44.44.44 -> 30.30.30.30 >>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN >>> fragment id 0x692e, flags DONT_FRAGMENT >>> TCP: 45723 -> 1234 >>> seq. 0x119d35d0 ack 0x00000000 >>> flags 0x02 SYN, tcp header: 40 bytes >>> window 64240, checksum 0xd6be >>> 00:03:26:875803: ip4-local >>> TCP: 44.44.44.44 -> 30.30.30.30 >>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN >>> fragment id 0x692e, flags DONT_FRAGMENT >>> TCP: 45723 -> 1234 >>> seq. 0x119d35d0 ack 0x00000000 >>> flags 0x02 SYN, tcp header: 40 bytes >>> window 64240, checksum 0xd6be >>> 00:03:26:875805: an_ppe_router_input >>> AN_PPE_ROUTER_INPUT: sw_if_index 1, next index 3 >>> >>> 00:03:26:875819: tcp4-input >>> [0:0][T] :::0->:::0 state CLOSED >>> TCP: 45723 -> 1234 >>> seq. 0x119d35d0 ack 0x00000000 >>> flags 0x02 SYN, tcp header: 40 bytes >>> window 64240, checksum 0xd6be >>> 00:03:26:875826: tcp4-listen >>> 1234 -> 45723 (LISTEN) >>> >>> >>> Interface details >>> ==================== >>> vpp# show interface >>> Name Idx State MTU (L3/IP4/IP6/MPLS) >>> Counter Count >>> VirtualFuncEthernet0/7/0 3 up 1500/0/0/0 rx >>> packets 3922 >>> rx >>> bytes 239610 >>> tx >>> packets 17 >>> tx >>> bytes 6704 >>> >>> drops 3914 >>> VirtualFuncEthernet0/7/0.1557 13 up 1500/0/0/0 >>> VirtualFuncEthernet0/7/0.1556 14 up 1500/0/0/0 rx >>> packets 22 >>> rx >>> bytes 5130 >>> tx >>> packets 17 >>> tx >>> bytes 6704 >>> >>> drops 14 >>> ip4 >>> 19 >>> VirtualFuncEthernet0/8/0 4 down 9000/0/0/0 >>> gre1 2 up 9000/0/0/0 >>> ipip0 1 up 9000/0/0/0 rx >>> packets 5 >>> rx >>> bytes 500 >>> ip4 >>> 5 >>> local0 0 down 0/0/0/0 >>> loop2 8 up 9000/0/0/0 >>> loop3 9 up 9000/0/0/0 >>> loop4 10 up 9000/0/0/0 >>> loop5 11 up 9000/0/0/0 >>> loop6 12 up 9000/0/0/0 >>> memif0/0 5 up 0/65535/0/0 rx >>> packets 1036 >>> rx >>> bytes 67766832 >>> tx >>> packets 1049 >>> tx >>> bytes 67822842 >>> >>> drops 2 >>> ip4 >>> 1036 >>> memif128/0 6 up 0/65535/0/0 >>> memif192/0 7 up 0/65535/0/0 >>> memif210/0 15 up 0/65535/0/0 >>> memif210/1 16 up 0/65535/0/0 >>> vpp# >>> vpp# >>> >>> On Tue, Mar 15, 2022 at 2:32 PM Vijay Kumar via lists.fd.io >>> <vjkumar2003=gmail....@lists.fd.io> wrote: >>> >>>> Hi experts, >>>> >>>> I recently integrated vpp stack 21.06 and am running my NAS TCP >>>> protocol test cases. The pcap dispatch trace that I captured shows the TCP >>>> SYN packets are coming to tcp-input() and then to tcp-listen() but from >>>> here pkts are dropped instead of going to tcp-output() graph nodes for >>>> sending the SYN-ACK >>>> >>>> NOTE >>>> ========= >>>> The session layer is enabled and the TCP listen IP/port is also showing >>>> in the session verbose command. >>>> >>>> The same test case used to work fine in the VPP 20.05. Am I missing >>>> anything? >>>> Let me know if there is anything I need to look at. >>>> >>>> vpp# show session verbose >>>> Connection State >>>> Rx-f Tx-f >>>> [0:0][T] 30.30.30.30:1234->0.0.0.0:0 LISTEN >>>> 0 0 >>>> Thread 0: active sessions 1 >>>> Thread 1: no sessions >>>> >>>> >>>> >>>> >>> >>> >>> >>> >> >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21034): https://lists.fd.io/g/vpp-dev/message/21034 Mute This Topic: https://lists.fd.io/mt/89793823/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
