Hi Florin, Thanks for the clarification about the TCP changes b/w the 2 releases
I will use your patch, hopefully I will catch the issue about where the drops are. I will try to debug further. I will revert back if required. Regards. On Wed, Mar 16, 2022 at 11:12 AM Florin Coras <fcoras.li...@gmail.com> wrote: > Hi Vijay, > > On Mar 15, 2022, at 9:58 PM, Vijay Kumar <vjkumar2...@gmail.com> wrote: > > Hi florin, > > Thanks a lot for helping me out. I will try your patch and update you with > the result. > > > Thanks! > > > > A general observation > ====================== > In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD > counter is incremented. > > > That’s what I’d expect as well. > > > I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph > node tcp46_listen_inline() of VPP 20.05 the SYN pkt trace function is > called towards the end i.e. after calling tcp_send_synack() but in 21.06 > code, the tcp46_listen_trace_frame() is called at the very beginning > of tcp46_listen_inline() graph node. > > I think moving the pkt trace macro to the end of the function is good > (placing it close to the line that calls tcp_send_synack), otherwise it can > mislead us and will not help debugging. > > > Most of the changes were code refactoring/cleanup and buffer handling > optimizations, not protocol related. TCP tracing, at least as it is now, > doesn’t provide info about the errors hit, instead it reports the > connections hit and the reporting of errors is done through node and > session counters (see show session verbose 2). Obviously that didn’t work > properly for the listen node. > > Regards, > Florin > > > > On Wed, Mar 16, 2022 at 10:22 AM Florin Coras <fcoras.li...@gmail.com> > wrote: > >> Hi Vijay, >> >> That’s probably because packets are hitting an actual error and it seems >> the listen node is not reporting anything but syns received. Here’s a patch >> that might help [1]. It might not cherry-pick cleanly on 21.06. >> >> Regards, >> Florin >> >> [1] https://gerrit.fd.io/r/c/vpp/+/35654 >> >> On Mar 15, 2022, at 7:54 PM, Vijay Kumar <vjkumar2...@gmail.com> wrote: >> >> Hi Florin, >> >> I have the output of show node counters (show errors) taken on both 20.05 >> and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or >> tcp4-output etc. >> >> Please let me know why SYN rcvd counter itself is not incremented but in >> the earlier reply I already pasted the show trace ouput where we saw the >> SYN pkt was landed on tcp4-litsen node. >> >> VPP 20.05 >> =========== >> pp# show node counters >> Count Node Reason >> 3 memif-input not ip packet >> 10259 an_ppe_wfectrl wfectrl packets received >> 10259 an_ppe_wfectrl wfectrl replies sent >> 1 an_ppe_wfectrl wfectrl packet >> processing failed >> 8123 an_ppe_wfectrl session stat request >> received >> 1 an_ppe_wfectrl service construct >> config request received >> 1 an_ppe_wfectrl service construct >> config request success >> 1 an_ppe_wfectrl service config request >> received >> 1 an_ppe_wfectrl service config request >> success >> 1686 an_ppe_wfectrl dpi stats request >> received >> 1686 an_ppe_wfectrl dpi stats request >> success >> 70 an_ppe_wfectrl nat stats request >> received >> 70 an_ppe_wfectrl nat stats request >> success >> 70 an_ppe_wfectrl vpp stats request >> received >> 70 an_ppe_wfectrl vpp stats request >> success >> 1 an_ppe_wfectrl udp tunnel resource >> block add request received >> 1 an_ppe_wfectrl udp tunnel resource >> block add request success >> 1 an_ppe_wfectrl l3rm resource block >> update request received >> 1 an_ppe_wfectrl l3rm resource block >> update request success >> 1 an_ppe_wfectrl ue registration request >> received >> 17 an_ppe_wfectrl ike msg received from >> ikemgr >> 17 an_ppe_wfectrl ike msg send to network >> success >> 3 an_ppe_wfectrl ipsec sa install msg >> received from ikemgr >> 3 an_ppe_wfectrl ipsec sa install msg >> processed successfully >> 1 an_ppe_vppctrl_input Success in insert in >> ueidentifier >> 1 an_ppe_vppctrl_input Success in insert in >> uecontext >> 2 an_ppe_vppctrl_input Downlink message sent >> to UE successfully >> 1 an_ppe_vppctrl_input TCP ESTABLISHMENT >> message received >> 1 an_ppe_vppctrl_input UPLINK NAS message >> received >> 1 an_ppe_vppctrl_input NAS TCP CLOSE message >> received >> 7 an_ppe_router_input packets received in rtr >> 7 an_ppe_router_input packets dropped to >> host. no tacptcp session found >> 17 an-ppe-isakmp4-output Total IKEV4 packets >> dispatched to network >> 17 an-ppe-isakmpmgr-input packets processed by >> isakmpmgr input plugin >> 17 an-ppe-isakmpmgr-input Received IKE packets >> 17 an-ppe-isakmpmgr-input Successfully sent ike >> message to Session Manager >> 5 an-ppe-isakmpmgr-input Received ike exchange >> AUTH packet >> 4 an-ppe-isakmpmgr-input Received ike exchange >> CREATE CHILD SA packet >> 1 an-ppe-isakmpmgr-input Received ike exchange >> SA INIT packet >> 7 an-ppe-isakmpmgr-input Received ike exchange >> INFORMATIONAL packet >> 5 arp-reply ARP replies sent >> 4 session-queue Packets transmitted >> 17 ip4-udp-lookup No error >> 1 tcp4-listen SYNs received >> 2 tcp4-rcv-process Pure ACKs received >> 1 tcp4-established Packets pushed into rx >> fifo >> 2 tcp4-established Pure ACKs received >> 1 tcp4-established FINs received >> 5 tcp4-output Packets sent >> 7 esp4-decrypt-tun ESP pkts received >> 5 esp4-encrypt-tun ESP pkts received >> 5 esp4-encrypt-tun total control pkts >> received >> 7 ipsec4-tun-input good packets received >> 2 ip4-local unknown ip protocol >> 5 ethernet-input unknown vlan >> vpp# >> vpp# >> >> >> >> >> >> VPP 21.06 output >> =============== >> vpp# >> vpp# show version >> vpp v21.06.0-3~g2a4af2b5d-dirty built by an-vijay_kumar on 500aaad2d090 >> at 2022-03-15T01:16:27 >> vpp# >> vpp# >> vpp# show node counters >> Count Node >> Reason Severity >> 1 null-node >> blackholed packets error >> 6690 an_ppe_wfectrl wfectrl >> packets received error >> 6690 an_ppe_wfectrl wfectrl >> replies sent error >> 1 an_ppe_wfectrl wfectrl packet >> processing failed error >> 5329 an_ppe_wfectrl session stat >> request received error >> 1 an_ppe_wfectrl service construct >> config request received error >> 1 an_ppe_wfectrl service construct >> config request success error >> 1 an_ppe_wfectrl service config >> request received error >> 1 an_ppe_wfectrl service config >> request success error >> 1068 an_ppe_wfectrl dpi stats >> request received error >> 1068 an_ppe_wfectrl dpi stats >> request success error >> 44 an_ppe_wfectrl nat stats >> request received error >> 44 an_ppe_wfectrl nat stats >> request success error >> 44 an_ppe_wfectrl vpp stats >> request received error >> 44 an_ppe_wfectrl vpp stats >> request success error >> 1 an_ppe_wfectrl udp tunnel resource >> block add request received error >> 1 an_ppe_wfectrl udp tunnel resource >> block add request success error >> 1 an_ppe_wfectrl l3rm resource block >> update request received error >> 1 an_ppe_wfectrl l3rm resource block >> update request success error >> 1 an_ppe_wfectrl ue registration >> request received error >> 1 an_ppe_wfectrl ue deregistration >> request received error >> 14 an_ppe_wfectrl ike msg >> received from ikemgr error >> 14 an_ppe_wfectrl ike msg send to >> network success error >> 3 an_ppe_wfectrl ipsec sa install msg >> received from ikemgr error >> 3 an_ppe_wfectrl ipsec sa delete msg >> received from ikemgr error >> 3 an_ppe_wfectrl ipsec sa install msg >> processed successfully error >> 3 an_ppe_wfectrl ipsec sa delete msg >> processed successfully error >> 1 an_ppe_vppctrl_input Success in delete >> in ueidentifier error >> 1 an_ppe_vppctrl_input Success in insert >> in ueidentifier error >> 1 an_ppe_vppctrl_input Success in >> insert in uecontext error >> 1 an_ppe_vppctrl_input Success in >> delete in uecontext error >> 5 an_ppe_router_input packets >> received in rtr error >> 5 an_ppe_router_input packets dropped to host. >> no tacptcp session found error >> 14 an-ppe-isakmp4-output Total IKEV4 packets >> dispatched to network error >> 14 an-ppe-isakmpmgr-input packets processed by >> isakmpmgr input plugin error >> 14 an-ppe-isakmpmgr-input Received >> IKE packets error >> 14 an-ppe-isakmpmgr-input Successfully sent ike >> message to Session Manager error >> 5 an-ppe-isakmpmgr-input Received ike >> exchange AUTH packet error >> 4 an-ppe-isakmpmgr-input Received ike exchange >> CREATE CHILD SA packet error >> 1 an-ppe-isakmpmgr-input Received ike >> exchange SA INIT packet error >> 4 an-ppe-isakmpmgr-input Received ike exchange >> INFORMATIONAL packet error >> 3 arp-reply ARP >> replies sent error >> 14 ip4-udp-lookup No >> error error >> 5 esp4-decrypt-tun ESP pkts >> received error >> 5 ipsec4-tun-input good >> packets received error >> 1 ip4-input ip4 >> ttl <= 1 error >> 1 ip4-icmp-error hop limit >> exceeded response sent error >> 27628 ethernet-input >> unknown vlan error >> vpp# >> vpp# >> >> >> >> >> On Wed, Mar 16, 2022 at 6:52 AM Florin Coras <fcoras.li...@gmail.com> >> wrote: >> >>> Hi Vijay, >>> >>> You won’t see the syn/syn-acks in traces because of the way they are >>> generated. Nonetheless, you can verify that the syns were received with >>> “show error” and check that the syn-acks were actually generated with a >>> pcap trace rxtx <output interface> >>> >>> Previously tracing worked because buffers were re-used and incoming >>> packets were directly sent to output. More recently we’ve started >>> dispatching syn-acks through the session layer in order to minimize size of >>> tx bursts per dispatch. >>> >>> Regards, >>> Florin >>> >>> On Mar 15, 2022, at 6:08 PM, Vijay Kumar <vjkumar2...@gmail.com> wrote: >>> >>> Hi Florin, >>> >>> Thanks for the quick response. >>> >>> It means it is expected to see the SYN pkts getting dropped after it is >>> terminated but I was not able to see any SYN-ACK going out of VPP. >>> >>> In both, "show trace" cmd output and also the pcap taken at dspatch >>> graph node level, I was not able to see the SYN-ACK going out. The route >>> to the SYN-ACK destination (which is the original source that started SYN) >>> is also present in the ip fib output. The configuration is the same that >>> was working fine for me in vp 20.05. >>> >>> Is there anything that I can look at for the SYN-ACK not sending issue? >>> >>> >>> >>> On Wed, 16 Mar 2022, 00:40 Florin Coras, <fcoras.li...@gmail.com> wrote: >>> >>>> Hi Vijay, >>>> >>>> I see an_ppe_router_input forwards syns to tcp-input and those packets >>>> are delivered to tcp-listen, i.e., you’ve created a listener that’s matched >>>> by the incoming traffic. >>>> >>>> The thing to keep in mind is that tcp terminates incoming flows and it >>>> does not reuse buffers. That is, the syn hits the listen node and a syn-ack >>>> is programmed for this new connection that still needs to complete the >>>> handshake. The original syn packet is discarded and therefore you see it as >>>> a drop. >>>> >>>> Regards, >>>> Florin >>>> >>>> On Mar 15, 2022, at 3:50 AM, Vijay Kumar <vjkumar2...@gmail.com> wrote: >>>> >>>> The is the output of show trace and show interface >>>> ============================================ >>>> >>>> Packet 36 >>>> >>>> 00:03:26:875694: dpdk-input >>>> VirtualFuncEthernet0/7/0 rx queue 0 >>>> buffer 0x4c1b21: current data 0, length 138, buffer-pool 0, ref-count >>>> 1, totlen-nifb 0, trace handle 0x1000023 >>>> ext-hdr-valid >>>> l4-cksum-computed l4-cksum-correct >>>> PKT MBUF: port 0, nb_segs 1, pkt_len 138 >>>> buf_len 2176, data_len 138, ol_flags 0x182, data_off 128, phys_addr >>>> 0xc266c8c0 >>>> packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 >>>> rss 0xde0c21da fdir.hi 0x0 fdir.lo 0xde0c21da >>>> Packet Offload Flags >>>> PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result >>>> PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid >>>> PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid >>>> Packet Types >>>> RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet >>>> RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or >>>> without extension headers >>>> RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet >>>> IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556 >>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>>> fragment id 0x0000, flags DONT_FRAGMENT >>>> 00:03:26:875702: ethernet-input >>>> frame: flags 0x3, hw-if-index 3, sw-if-index 3 >>>> IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556 >>>> 00:03:26:875708: ip4-input >>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>>> fragment id 0x0000, flags DONT_FRAGMENT >>>> 00:03:26:875714: ip4-lookup >>>> fib 2 dpo-idx 27 flow hash: 0x00000000 >>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>>> fragment id 0x0000, flags DONT_FRAGMENT >>>> 00:03:26:875715: ip4-local >>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80 >>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN >>>> fragment id 0x0000, flags DONT_FRAGMENT >>>> 00:03:26:875717: ipsec4-tun-input >>>> IPSec: remote:20.20.99.215 spi:4213 (0x00001075) sa:1 tun:0 seq 1 sa >>>> -2029505104 >>>> 00:03:26:875719: esp4-decrypt-tun >>>> esp: crypto aes-cbc-256 integrity sha1-96 pkt-seq 1 sa-seq 1 >>>> sa-seq-hi 0 >>>> 00:03:26:875801: ip4-input-no-checksum >>>> TCP: 44.44.44.44 -> 30.30.30.30 >>>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN >>>> fragment id 0x692e, flags DONT_FRAGMENT >>>> TCP: 45723 -> 1234 >>>> seq. 0x119d35d0 ack 0x00000000 >>>> flags 0x02 SYN, tcp header: 40 bytes >>>> window 64240, checksum 0xd6be >>>> 00:03:26:875803: ip4-lookup >>>> fib 2 dpo-idx 25 flow hash: 0x00000000 >>>> TCP: 44.44.44.44 -> 30.30.30.30 >>>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN >>>> fragment id 0x692e, flags DONT_FRAGMENT >>>> TCP: 45723 -> 1234 >>>> seq. 0x119d35d0 ack 0x00000000 >>>> flags 0x02 SYN, tcp header: 40 bytes >>>> window 64240, checksum 0xd6be >>>> 00:03:26:875803: ip4-local >>>> TCP: 44.44.44.44 -> 30.30.30.30 >>>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN >>>> fragment id 0x692e, flags DONT_FRAGMENT >>>> TCP: 45723 -> 1234 >>>> seq. 0x119d35d0 ack 0x00000000 >>>> flags 0x02 SYN, tcp header: 40 bytes >>>> window 64240, checksum 0xd6be >>>> 00:03:26:875805: an_ppe_router_input >>>> AN_PPE_ROUTER_INPUT: sw_if_index 1, next index 3 >>>> >>>> 00:03:26:875819: tcp4-input >>>> [0:0][T] :::0->:::0 state CLOSED >>>> TCP: 45723 -> 1234 >>>> seq. 0x119d35d0 ack 0x00000000 >>>> flags 0x02 SYN, tcp header: 40 bytes >>>> window 64240, checksum 0xd6be >>>> 00:03:26:875826: tcp4-listen >>>> 1234 -> 45723 (LISTEN) >>>> >>>> >>>> Interface details >>>> ==================== >>>> vpp# show interface >>>> Name Idx State MTU (L3/IP4/IP6/MPLS) >>>> Counter Count >>>> VirtualFuncEthernet0/7/0 3 up 1500/0/0/0 rx >>>> packets 3922 >>>> rx >>>> bytes 239610 >>>> tx >>>> packets 17 >>>> tx >>>> bytes 6704 >>>> >>>> drops 3914 >>>> VirtualFuncEthernet0/7/0.1557 13 up 1500/0/0/0 >>>> VirtualFuncEthernet0/7/0.1556 14 up 1500/0/0/0 rx >>>> packets 22 >>>> rx >>>> bytes 5130 >>>> tx >>>> packets 17 >>>> tx >>>> bytes 6704 >>>> >>>> drops 14 >>>> ip4 >>>> 19 >>>> VirtualFuncEthernet0/8/0 4 down 9000/0/0/0 >>>> gre1 2 up 9000/0/0/0 >>>> ipip0 1 up 9000/0/0/0 rx >>>> packets 5 >>>> rx >>>> bytes 500 >>>> ip4 >>>> 5 >>>> local0 0 down 0/0/0/0 >>>> loop2 8 up 9000/0/0/0 >>>> loop3 9 up 9000/0/0/0 >>>> loop4 10 up 9000/0/0/0 >>>> loop5 11 up 9000/0/0/0 >>>> loop6 12 up 9000/0/0/0 >>>> memif0/0 5 up 0/65535/0/0 rx >>>> packets 1036 >>>> rx >>>> bytes 67766832 >>>> tx >>>> packets 1049 >>>> tx >>>> bytes 67822842 >>>> >>>> drops 2 >>>> ip4 >>>> 1036 >>>> memif128/0 6 up 0/65535/0/0 >>>> memif192/0 7 up 0/65535/0/0 >>>> memif210/0 15 up 0/65535/0/0 >>>> memif210/1 16 up 0/65535/0/0 >>>> vpp# >>>> vpp# >>>> >>>> On Tue, Mar 15, 2022 at 2:32 PM Vijay Kumar via lists.fd.io >>>> <vjkumar2003=gmail....@lists.fd.io> wrote: >>>> >>>>> Hi experts, >>>>> >>>>> I recently integrated vpp stack 21.06 and am running my NAS TCP >>>>> protocol test cases. The pcap dispatch trace that I captured shows the >>>>> TCP >>>>> SYN packets are coming to tcp-input() and then to tcp-listen() but from >>>>> here pkts are dropped instead of going to tcp-output() graph nodes for >>>>> sending the SYN-ACK >>>>> >>>>> NOTE >>>>> ========= >>>>> The session layer is enabled and the TCP listen IP/port is also >>>>> showing in the session verbose command. >>>>> >>>>> The same test case used to work fine in the VPP 20.05. Am I missing >>>>> anything? >>>>> Let me know if there is anything I need to look at. >>>>> >>>>> vpp# show session verbose >>>>> Connection State >>>>> Rx-f Tx-f >>>>> [0:0][T] 30.30.30.30:1234->0.0.0.0:0 LISTEN >>>>> 0 0 >>>>> Thread 0: active sessions 1 >>>>> Thread 1: no sessions >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >> >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21036): https://lists.fd.io/g/vpp-dev/message/21036 Mute This Topic: https://lists.fd.io/mt/89793823/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
