Hi Vijay, > On Mar 15, 2022, at 9:58 PM, Vijay Kumar <vjkumar2...@gmail.com> wrote: > > Hi florin, > > Thanks a lot for helping me out. I will try your patch and update you with > the result.
Thanks! > > > A general observation > ====================== > In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD counter > is incremented. That’s what I’d expect as well. > > I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph node > tcp46_listen_inline() of VPP 20.05 the SYN pkt trace function is called > towards the end i.e. after calling tcp_send_synack() but in 21.06 code, the > tcp46_listen_trace_frame() is called at the very beginning of > tcp46_listen_inline() graph node. > > I think moving the pkt trace macro to the end of the function is good > (placing it close to the line that calls tcp_send_synack), otherwise it can > mislead us and will not help debugging. > Most of the changes were code refactoring/cleanup and buffer handling optimizations, not protocol related. TCP tracing, at least as it is now, doesn’t provide info about the errors hit, instead it reports the connections hit and the reporting of errors is done through node and session counters (see show session verbose 2). Obviously that didn’t work properly for the listen node. Regards, Florin > > > On Wed, Mar 16, 2022 at 10:22 AM Florin Coras <fcoras.li...@gmail.com > <mailto:fcoras.li...@gmail.com>> wrote: > Hi Vijay, > > That’s probably because packets are hitting an actual error and it seems the > listen node is not reporting anything but syns received. Here’s a patch that > might help [1]. It might not cherry-pick cleanly on 21.06. > > Regards, > Florin > > [1] https://gerrit.fd.io/r/c/vpp/+/35654 > <https://gerrit.fd.io/r/c/vpp/+/35654> > >> On Mar 15, 2022, at 7:54 PM, Vijay Kumar <vjkumar2...@gmail.com >> <mailto:vjkumar2...@gmail.com>> wrote: >> >> Hi Florin, >> >> I have the output of show node counters (show errors) taken on both 20.05 >> and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or >> tcp4-output etc. >> >> Please let me know why SYN rcvd counter itself is not incremented but in the >> earlier reply I already pasted the show trace ouput where we saw the SYN pkt >> was landed on tcp4-litsen node. >> >> VPP 20.05 >> =========== >> pp# show node counters >> Count Node Reason >> 3 memif-input not ip packet >> 10259 an_ppe_wfectrl wfectrl packets received >> 10259 an_ppe_wfectrl wfectrl replies sent >> 1 an_ppe_wfectrl wfectrl packet processing >> failed >> 8123 an_ppe_wfectrl session stat request >> received >> 1 an_ppe_wfectrl service construct config >> request received >> 1 an_ppe_wfectrl service construct config >> request success >> 1 an_ppe_wfectrl service config request >> received >> 1 an_ppe_wfectrl service config request >> success >> 1686 an_ppe_wfectrl dpi stats request received >> 1686 an_ppe_wfectrl dpi stats request success >> 70 an_ppe_wfectrl nat stats request received >> 70 an_ppe_wfectrl nat stats request success >> 70 an_ppe_wfectrl vpp stats request received >> 70 an_ppe_wfectrl vpp stats request success >> 1 an_ppe_wfectrl udp tunnel resource block >> add request received >> 1 an_ppe_wfectrl udp tunnel resource block >> add request success >> 1 an_ppe_wfectrl l3rm resource block update >> request received >> 1 an_ppe_wfectrl l3rm resource block update >> request success >> 1 an_ppe_wfectrl ue registration request >> received >> 17 an_ppe_wfectrl ike msg received from >> ikemgr >> 17 an_ppe_wfectrl ike msg send to network >> success >> 3 an_ppe_wfectrl ipsec sa install msg >> received from ikemgr >> 3 an_ppe_wfectrl ipsec sa install msg >> processed successfully >> 1 an_ppe_vppctrl_input Success in insert in >> ueidentifier >> 1 an_ppe_vppctrl_input Success in insert in >> uecontext >> 2 an_ppe_vppctrl_input Downlink message sent to >> UE successfully >> 1 an_ppe_vppctrl_input TCP ESTABLISHMENT message >> received >> 1 an_ppe_vppctrl_input UPLINK NAS message received >> 1 an_ppe_vppctrl_input NAS TCP CLOSE message >> received >> 7 an_ppe_router_input packets received in rtr >> 7 an_ppe_router_input packets dropped to host. >> no tacptcp session found >> 17 an-ppe-isakmp4-output Total IKEV4 packets >> dispatched to network >> 17 an-ppe-isakmpmgr-input packets processed by >> isakmpmgr input plugin >> 17 an-ppe-isakmpmgr-input Received IKE packets >> 17 an-ppe-isakmpmgr-input Successfully sent ike >> message to Session Manager >> 5 an-ppe-isakmpmgr-input Received ike exchange AUTH >> packet >> 4 an-ppe-isakmpmgr-input Received ike exchange >> CREATE CHILD SA packet >> 1 an-ppe-isakmpmgr-input Received ike exchange SA >> INIT packet >> 7 an-ppe-isakmpmgr-input Received ike exchange >> INFORMATIONAL packet >> 5 arp-reply ARP replies sent >> 4 session-queue Packets transmitted >> 17 ip4-udp-lookup No error >> 1 tcp4-listen SYNs received >> 2 tcp4-rcv-process Pure ACKs received >> 1 tcp4-established Packets pushed into rx fifo >> 2 tcp4-established Pure ACKs received >> 1 tcp4-established FINs received >> 5 tcp4-output Packets sent >> 7 esp4-decrypt-tun ESP pkts received >> 5 esp4-encrypt-tun ESP pkts received >> 5 esp4-encrypt-tun total control pkts received >> 7 ipsec4-tun-input good packets received >> 2 ip4-local unknown ip protocol >> 5 ethernet-input unknown vlan >> vpp# >> vpp# >> >> >> >> >> >> VPP 21.06 output >> =============== >> vpp# >> vpp# show version >> vpp v21.06.0-3~g2a4af2b5d-dirty built by an-vijay_kumar on 500aaad2d090 at >> 2022-03-15T01:16:27 >> vpp# >> vpp# >> vpp# show node counters >> Count Node Reason >> Severity >> 1 null-node blackholed >> packets error >> 6690 an_ppe_wfectrl wfectrl packets >> received error >> 6690 an_ppe_wfectrl wfectrl >> replies sent error >> 1 an_ppe_wfectrl wfectrl packet >> processing failed error >> 5329 an_ppe_wfectrl session stat >> request received error >> 1 an_ppe_wfectrl service construct config >> request received error >> 1 an_ppe_wfectrl service construct >> config request success error >> 1 an_ppe_wfectrl service config >> request received error >> 1 an_ppe_wfectrl service config >> request success error >> 1068 an_ppe_wfectrl dpi stats >> request received error >> 1068 an_ppe_wfectrl dpi stats >> request success error >> 44 an_ppe_wfectrl nat stats >> request received error >> 44 an_ppe_wfectrl nat stats >> request success error >> 44 an_ppe_wfectrl vpp stats >> request received error >> 44 an_ppe_wfectrl vpp stats >> request success error >> 1 an_ppe_wfectrl udp tunnel resource block >> add request received error >> 1 an_ppe_wfectrl udp tunnel resource block >> add request success error >> 1 an_ppe_wfectrl l3rm resource block >> update request received error >> 1 an_ppe_wfectrl l3rm resource block >> update request success error >> 1 an_ppe_wfectrl ue registration >> request received error >> 1 an_ppe_wfectrl ue deregistration >> request received error >> 14 an_ppe_wfectrl ike msg received >> from ikemgr error >> 14 an_ppe_wfectrl ike msg send to >> network success error >> 3 an_ppe_wfectrl ipsec sa install msg >> received from ikemgr error >> 3 an_ppe_wfectrl ipsec sa delete msg >> received from ikemgr error >> 3 an_ppe_wfectrl ipsec sa install msg >> processed successfully error >> 3 an_ppe_wfectrl ipsec sa delete msg >> processed successfully error >> 1 an_ppe_vppctrl_input Success in delete >> in ueidentifier error >> 1 an_ppe_vppctrl_input Success in insert in >> ueidentifier error >> 1 an_ppe_vppctrl_input Success in insert >> in uecontext error >> 1 an_ppe_vppctrl_input Success in delete >> in uecontext error >> 5 an_ppe_router_input packets >> received in rtr error >> 5 an_ppe_router_input packets dropped to host. no >> tacptcp session found error >> 14 an-ppe-isakmp4-output Total IKEV4 packets >> dispatched to network error >> 14 an-ppe-isakmpmgr-input packets processed by >> isakmpmgr input plugin error >> 14 an-ppe-isakmpmgr-input Received IKE >> packets error >> 14 an-ppe-isakmpmgr-input Successfully sent ike >> message to Session Manager error >> 5 an-ppe-isakmpmgr-input Received ike >> exchange AUTH packet error >> 4 an-ppe-isakmpmgr-input Received ike exchange >> CREATE CHILD SA packet error >> 1 an-ppe-isakmpmgr-input Received ike exchange >> SA INIT packet error >> 4 an-ppe-isakmpmgr-input Received ike exchange >> INFORMATIONAL packet error >> 3 arp-reply ARP replies >> sent error >> 14 ip4-udp-lookup No >> error error >> 5 esp4-decrypt-tun ESP pkts >> received error >> 5 ipsec4-tun-input good packets >> received error >> 1 ip4-input ip4 ttl >> <= 1 error >> 1 ip4-icmp-error hop limit exceeded >> response sent error >> 27628 ethernet-input unknown >> vlan error >> vpp# >> vpp# >> >> >> >> >> On Wed, Mar 16, 2022 at 6:52 AM Florin Coras <fcoras.li...@gmail.com >> <mailto:fcoras.li...@gmail.com>> wrote: >> Hi Vijay, >> >> You won’t see the syn/syn-acks in traces because of the way they are >> generated. Nonetheless, you can verify that the syns were received with >> “show error” and check that the syn-acks were actually generated with a pcap >> trace rxtx <output interface> >> >> Previously tracing worked because buffers were re-used and incoming packets >> were directly sent to output. More recently we’ve started dispatching >> syn-acks through the session layer in order to minimize size of tx bursts >> per dispatch. >> >> Regards, >> Florin >> >>> On Mar 15, 2022, at 6:08 PM, Vijay Kumar <vjkumar2...@gmail.com >>> <mailto:vjkumar2...@gmail.com>> wrote: >>> >>> Hi Florin, >>> >>> Thanks for the quick response. >>> >>> It means it is expected to see the SYN pkts getting dropped after it is >>> terminated but I was not able to see any SYN-ACK going out of VPP. >>> >>> In both, "show trace" cmd output and also the pcap taken at dspatch graph >>> node level, I was not able to see the SYN-ACK going out. The route to the >>> SYN-ACK destination (which is the original source that started SYN) is also >>> present in the ip fib output. The configuration is the same that was >>> working fine for me in vp 20.05. >>> >>> Is there anything that I can look at for the SYN-ACK not sending issue? >>> >>> >>> >>> On Wed, 16 Mar 2022, 00:40 Florin Coras, <fcoras.li...@gmail.com >>> <mailto:fcoras.li...@gmail.com>> wrote: >>> Hi Vijay, >>> >>> I see an_ppe_router_input forwards syns to tcp-input and those packets are >>> delivered to tcp-listen, i.e., you’ve created a listener that’s matched by >>> the incoming traffic. >>> >>> The thing to keep in mind is that tcp terminates incoming flows and it does >>> not reuse buffers. That is, the syn hits the listen node and a syn-ack is >>> programmed for this new connection that still needs to complete the >>> handshake. The original syn packet is discarded and therefore you see it as >>> a drop. >>> >>> Regards, >>> Florin >>> >>>> On Mar 15, 2022, at 3:50 AM, Vijay Kumar <vjkumar2...@gmail.com >>>> <mailto:vjkumar2...@gmail.com>> wrote: >>>> >>>> The is the output of show trace and show interface >>>> ============================================ >>>> >>>> Packet 36 >>>> >>>> 00:03:26:875694: dpdk-input >>>> VirtualFuncEthernet0/7/0 rx queue 0 >>>> buffer 0x4c1b21: current data 0, length 138, buffer-pool 0, ref-count 1, >>>> totlen-nifb 0, trace handle 0x1000023 >>>> ext-hdr-valid >>>> l4-cksum-computed l4-cksum-correct >>>> PKT MBUF: port 0, nb_segs 1, pkt_len 138 >>>> buf_len 2176, data_len 138, ol_flags 0x182, data_off 128, phys_addr >>>> 0xc266c8c0 >>>> packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 >>>> rss 0xde0c21da fdir.hi 0x0 fdir.lo 0xde0c21da >>>> Packet Offload Flags >>>> PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result >>>> PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid >>>> PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid >>>> Packet Types >>>> RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet >>>> RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without >>>> extension headers >>>> RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet >>>> IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556 >>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>>> fragment id 0x0000, flags DONT_FRAGMENT >>>> 00:03:26:875702: ethernet-input >>>> frame: flags 0x3, hw-if-index 3, sw-if-index 3 >>>> IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556 >>>> 00:03:26:875708: ip4-input >>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>>> fragment id 0x0000, flags DONT_FRAGMENT >>>> 00:03:26:875714: ip4-lookup >>>> fib 2 dpo-idx 27 flow hash: 0x00000000 >>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>>> fragment id 0x0000, flags DONT_FRAGMENT >>>> 00:03:26:875715: ip4-local >>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>>> fragment id 0x0000, flags DONT_FRAGMENT >>>> 00:03:26:875717: ipsec4-tun-input >>>> IPSec: remote:20.20.99.215 spi:4213 (0x00001075) sa:1 tun:0 seq 1 sa >>>> -2029505104 >>>> 00:03:26:875719: esp4-decrypt-tun >>>> esp: crypto aes-cbc-256 integrity sha1-96 pkt-seq 1 sa-seq 1 sa-seq-hi 0 >>>> 00:03:26:875801: ip4-input-no-checksum >>>> TCP: 44.44.44.44 -> 30.30.30.30 >>>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN >>>> fragment id 0x692e, flags DONT_FRAGMENT >>>> TCP: 45723 -> 1234 >>>> seq. 0x119d35d0 ack 0x00000000 >>>> flags 0x02 SYN, tcp header: 40 bytes >>>> window 64240, checksum 0xd6be >>>> 00:03:26:875803: ip4-lookup >>>> fib 2 dpo-idx 25 flow hash: 0x00000000 >>>> TCP: 44.44.44.44 -> 30.30.30.30 >>>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN >>>> fragment id 0x692e, flags DONT_FRAGMENT >>>> TCP: 45723 -> 1234 >>>> seq. 0x119d35d0 ack 0x00000000 >>>> flags 0x02 SYN, tcp header: 40 bytes >>>> window 64240, checksum 0xd6be >>>> 00:03:26:875803: ip4-local >>>> TCP: 44.44.44.44 -> 30.30.30.30 >>>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN >>>> fragment id 0x692e, flags DONT_FRAGMENT >>>> TCP: 45723 -> 1234 >>>> seq. 0x119d35d0 ack 0x00000000 >>>> flags 0x02 SYN, tcp header: 40 bytes >>>> window 64240, checksum 0xd6be >>>> 00:03:26:875805: an_ppe_router_input >>>> AN_PPE_ROUTER_INPUT: sw_if_index 1, next index 3 >>>> >>>> 00:03:26:875819: tcp4-input >>>> [0:0][T] :::0->:::0 state CLOSED >>>> TCP: 45723 -> 1234 >>>> seq. 0x119d35d0 ack 0x00000000 >>>> flags 0x02 SYN, tcp header: 40 bytes >>>> window 64240, checksum 0xd6be >>>> 00:03:26:875826: tcp4-listen >>>> 1234 -> 45723 (LISTEN) >>>> >>>> >>>> Interface details >>>> ==================== >>>> vpp# show interface >>>> Name Idx State MTU (L3/IP4/IP6/MPLS) >>>> Counter Count >>>> VirtualFuncEthernet0/7/0 3 up 1500/0/0/0 rx >>>> packets 3922 >>>> rx >>>> bytes 239610 >>>> tx >>>> packets 17 >>>> tx >>>> bytes 6704 >>>> drops >>>> 3914 >>>> VirtualFuncEthernet0/7/0.1557 13 up 1500/0/0/0 >>>> VirtualFuncEthernet0/7/0.1556 14 up 1500/0/0/0 rx >>>> packets 22 >>>> rx >>>> bytes 5130 >>>> tx >>>> packets 17 >>>> tx >>>> bytes 6704 >>>> drops >>>> 14 >>>> ip4 >>>> 19 >>>> VirtualFuncEthernet0/8/0 4 down 9000/0/0/0 >>>> gre1 2 up 9000/0/0/0 >>>> ipip0 1 up 9000/0/0/0 rx >>>> packets 5 >>>> rx >>>> bytes 500 >>>> ip4 >>>> 5 >>>> local0 0 down 0/0/0/0 >>>> loop2 8 up 9000/0/0/0 >>>> loop3 9 up 9000/0/0/0 >>>> loop4 10 up 9000/0/0/0 >>>> loop5 11 up 9000/0/0/0 >>>> loop6 12 up 9000/0/0/0 >>>> memif0/0 5 up 0/65535/0/0 rx >>>> packets 1036 >>>> rx >>>> bytes 67766832 >>>> tx >>>> packets 1049 >>>> tx >>>> bytes 67822842 >>>> drops >>>> 2 >>>> ip4 >>>> 1036 >>>> memif128/0 6 up 0/65535/0/0 >>>> memif192/0 7 up 0/65535/0/0 >>>> memif210/0 15 up 0/65535/0/0 >>>> memif210/1 16 up 0/65535/0/0 >>>> vpp# >>>> vpp# >>>> >>>> On Tue, Mar 15, 2022 at 2:32 PM Vijay Kumar via lists.fd.io >>>> <http://lists.fd.io/> <vjkumar2003=gmail....@lists.fd.io >>>> <mailto:gmail....@lists.fd.io>> wrote: >>>> Hi experts, >>>> >>>> I recently integrated vpp stack 21.06 and am running my NAS TCP protocol >>>> test cases. The pcap dispatch trace that I captured shows the TCP SYN >>>> packets are coming to tcp-input() and then to tcp-listen() but from here >>>> pkts are dropped instead of going to tcp-output() graph nodes for sending >>>> the SYN-ACK >>>> >>>> NOTE >>>> ========= >>>> The session layer is enabled and the TCP listen IP/port is also showing in >>>> the session verbose command. >>>> >>>> The same test case used to work fine in the VPP 20.05. Am I missing >>>> anything? >>>> Let me know if there is anything I need to look at. >>>> >>>> vpp# show session verbose >>>> Connection State >>>> Rx-f Tx-f >>>> [0:0][T] 30.30.30.30:1234->0.0.0.0:0 <http://0.0.0.0:0/> >>>> LISTEN 0 0 >>>> Thread 0: active sessions 1 >>>> Thread 1: no sessions >>>> >>>> >>>> >>>> >>>> >>>> >>> >> >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21035): https://lists.fd.io/g/vpp-dev/message/21035 Mute This Topic: https://lists.fd.io/mt/89793823/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
