Hi Florin,
The patch helped me to find the exact point of failure in
tcp46_listen_inline() graph node.
When I tested again, I get the below error and it maps to this error code "
TCP_ERROR_CREATE_SESSION_FAIL"
This counter is incremented when the call to function
session_stream_accept() returns failure.
Is there any potential reason why allocation fails at this place?
show errors output
===============
4 arp-reply ARP replies
sent error
14 ip4-udp-lookup No
error error
5 tcp4-listen Sessions couldn't
be allocated error
5 esp4-decrypt-tun ESP pkts
received error
5 ipsec4-tun-input good packets
received error
1 ip4-input ip4 ttl
<= 1 error
1 ip4-icmp-error hop limit exceeded
response sent error
3346 ethernet-input unknown
vlan error
On Wed, Mar 16, 2022 at 1:31 PM Vijay Kumar <[email protected]> wrote:
> Hi Florin,
>
> Thanks for the clarification about the TCP changes b/w the 2 releases
>
> I will use your patch, hopefully I will catch the issue about where the
> drops are. I will try to debug further.
> I will revert back if required.
>
>
> Regards.
>
> On Wed, Mar 16, 2022 at 11:12 AM Florin Coras <[email protected]>
> wrote:
>
>> Hi Vijay,
>>
>> On Mar 15, 2022, at 9:58 PM, Vijay Kumar <[email protected]> wrote:
>>
>> Hi florin,
>>
>> Thanks a lot for helping me out. I will try your patch and update you
>> with the result.
>>
>>
>> Thanks!
>>
>>
>>
>> A general observation
>> ======================
>> In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD
>> counter is incremented.
>>
>>
>> That’s what I’d expect as well.
>>
>>
>> I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph
>> node tcp46_listen_inline() of VPP 20.05 the SYN pkt trace function is
>> called towards the end i.e. after calling tcp_send_synack() but in 21.06
>> code, the tcp46_listen_trace_frame() is called at the very beginning
>> of tcp46_listen_inline() graph node.
>>
>> I think moving the pkt trace macro to the end of the function is good
>> (placing it close to the line that calls tcp_send_synack), otherwise it can
>> mislead us and will not help debugging.
>>
>>
>> Most of the changes were code refactoring/cleanup and buffer handling
>> optimizations, not protocol related. TCP tracing, at least as it is now,
>> doesn’t provide info about the errors hit, instead it reports the
>> connections hit and the reporting of errors is done through node and
>> session counters (see show session verbose 2). Obviously that didn’t work
>> properly for the listen node.
>>
>> Regards,
>> Florin
>>
>>
>>
>> On Wed, Mar 16, 2022 at 10:22 AM Florin Coras <[email protected]>
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> That’s probably because packets are hitting an actual error and it seems
>>> the listen node is not reporting anything but syns received. Here’s a patch
>>> that might help [1]. It might not cherry-pick cleanly on 21.06.
>>>
>>> Regards,
>>> Florin
>>>
>>> [1] https://gerrit.fd.io/r/c/vpp/+/35654
>>>
>>> On Mar 15, 2022, at 7:54 PM, Vijay Kumar <[email protected]> wrote:
>>>
>>> Hi Florin,
>>>
>>> I have the output of show node counters (show errors) taken on both
>>> 20.05 and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or
>>> tcp4-output etc.
>>>
>>> Please let me know why SYN rcvd counter itself is not incremented but in
>>> the earlier reply I already pasted the show trace ouput where we saw the
>>> SYN pkt was landed on tcp4-litsen node.
>>>
>>> VPP 20.05
>>> ===========
>>> pp# show node counters
>>> Count Node Reason
>>> 3 memif-input not ip packet
>>> 10259 an_ppe_wfectrl wfectrl packets
>>> received
>>> 10259 an_ppe_wfectrl wfectrl replies sent
>>> 1 an_ppe_wfectrl wfectrl packet
>>> processing failed
>>> 8123 an_ppe_wfectrl session stat request
>>> received
>>> 1 an_ppe_wfectrl service construct
>>> config request received
>>> 1 an_ppe_wfectrl service construct
>>> config request success
>>> 1 an_ppe_wfectrl service config request
>>> received
>>> 1 an_ppe_wfectrl service config request
>>> success
>>> 1686 an_ppe_wfectrl dpi stats request
>>> received
>>> 1686 an_ppe_wfectrl dpi stats request
>>> success
>>> 70 an_ppe_wfectrl nat stats request
>>> received
>>> 70 an_ppe_wfectrl nat stats request
>>> success
>>> 70 an_ppe_wfectrl vpp stats request
>>> received
>>> 70 an_ppe_wfectrl vpp stats request
>>> success
>>> 1 an_ppe_wfectrl udp tunnel resource
>>> block add request received
>>> 1 an_ppe_wfectrl udp tunnel resource
>>> block add request success
>>> 1 an_ppe_wfectrl l3rm resource block
>>> update request received
>>> 1 an_ppe_wfectrl l3rm resource block
>>> update request success
>>> 1 an_ppe_wfectrl ue registration
>>> request received
>>> 17 an_ppe_wfectrl ike msg received from
>>> ikemgr
>>> 17 an_ppe_wfectrl ike msg send to
>>> network success
>>> 3 an_ppe_wfectrl ipsec sa install msg
>>> received from ikemgr
>>> 3 an_ppe_wfectrl ipsec sa install msg
>>> processed successfully
>>> 1 an_ppe_vppctrl_input Success in insert in
>>> ueidentifier
>>> 1 an_ppe_vppctrl_input Success in insert in
>>> uecontext
>>> 2 an_ppe_vppctrl_input Downlink message sent
>>> to UE successfully
>>> 1 an_ppe_vppctrl_input TCP ESTABLISHMENT
>>> message received
>>> 1 an_ppe_vppctrl_input UPLINK NAS message
>>> received
>>> 1 an_ppe_vppctrl_input NAS TCP CLOSE message
>>> received
>>> 7 an_ppe_router_input packets received in rtr
>>> 7 an_ppe_router_input packets dropped to
>>> host. no tacptcp session found
>>> 17 an-ppe-isakmp4-output Total IKEV4 packets
>>> dispatched to network
>>> 17 an-ppe-isakmpmgr-input packets processed by
>>> isakmpmgr input plugin
>>> 17 an-ppe-isakmpmgr-input Received IKE packets
>>> 17 an-ppe-isakmpmgr-input Successfully sent ike
>>> message to Session Manager
>>> 5 an-ppe-isakmpmgr-input Received ike exchange
>>> AUTH packet
>>> 4 an-ppe-isakmpmgr-input Received ike exchange
>>> CREATE CHILD SA packet
>>> 1 an-ppe-isakmpmgr-input Received ike exchange
>>> SA INIT packet
>>> 7 an-ppe-isakmpmgr-input Received ike exchange
>>> INFORMATIONAL packet
>>> 5 arp-reply ARP replies sent
>>> 4 session-queue Packets transmitted
>>> 17 ip4-udp-lookup No error
>>> 1 tcp4-listen SYNs received
>>> 2 tcp4-rcv-process Pure ACKs received
>>> 1 tcp4-established Packets pushed into rx
>>> fifo
>>> 2 tcp4-established Pure ACKs received
>>> 1 tcp4-established FINs received
>>> 5 tcp4-output Packets sent
>>> 7 esp4-decrypt-tun ESP pkts received
>>> 5 esp4-encrypt-tun ESP pkts received
>>> 5 esp4-encrypt-tun total control pkts
>>> received
>>> 7 ipsec4-tun-input good packets received
>>> 2 ip4-local unknown ip protocol
>>> 5 ethernet-input unknown vlan
>>> vpp#
>>> vpp#
>>>
>>>
>>>
>>>
>>>
>>> VPP 21.06 output
>>> ===============
>>> vpp#
>>> vpp# show version
>>> vpp v21.06.0-3~g2a4af2b5d-dirty built by an-vijay_kumar on 500aaad2d090
>>> at 2022-03-15T01:16:27
>>> vpp#
>>> vpp#
>>> vpp# show node counters
>>> Count Node
>>> Reason Severity
>>> 1 null-node
>>> blackholed packets error
>>> 6690 an_ppe_wfectrl wfectrl
>>> packets received error
>>> 6690 an_ppe_wfectrl wfectrl
>>> replies sent error
>>> 1 an_ppe_wfectrl wfectrl packet
>>> processing failed error
>>> 5329 an_ppe_wfectrl session stat
>>> request received error
>>> 1 an_ppe_wfectrl service construct
>>> config request received error
>>> 1 an_ppe_wfectrl service construct
>>> config request success error
>>> 1 an_ppe_wfectrl service config
>>> request received error
>>> 1 an_ppe_wfectrl service config
>>> request success error
>>> 1068 an_ppe_wfectrl dpi stats
>>> request received error
>>> 1068 an_ppe_wfectrl dpi stats
>>> request success error
>>> 44 an_ppe_wfectrl nat stats
>>> request received error
>>> 44 an_ppe_wfectrl nat stats
>>> request success error
>>> 44 an_ppe_wfectrl vpp stats
>>> request received error
>>> 44 an_ppe_wfectrl vpp stats
>>> request success error
>>> 1 an_ppe_wfectrl udp tunnel resource
>>> block add request received error
>>> 1 an_ppe_wfectrl udp tunnel resource
>>> block add request success error
>>> 1 an_ppe_wfectrl l3rm resource block
>>> update request received error
>>> 1 an_ppe_wfectrl l3rm resource block
>>> update request success error
>>> 1 an_ppe_wfectrl ue registration
>>> request received error
>>> 1 an_ppe_wfectrl ue
>>> deregistration request received error
>>> 14 an_ppe_wfectrl ike msg
>>> received from ikemgr error
>>> 14 an_ppe_wfectrl ike msg send to
>>> network success error
>>> 3 an_ppe_wfectrl ipsec sa install msg
>>> received from ikemgr error
>>> 3 an_ppe_wfectrl ipsec sa delete msg
>>> received from ikemgr error
>>> 3 an_ppe_wfectrl ipsec sa install msg
>>> processed successfully error
>>> 3 an_ppe_wfectrl ipsec sa delete msg
>>> processed successfully error
>>> 1 an_ppe_vppctrl_input Success in
>>> delete in ueidentifier error
>>> 1 an_ppe_vppctrl_input Success in
>>> insert in ueidentifier error
>>> 1 an_ppe_vppctrl_input Success in
>>> insert in uecontext error
>>> 1 an_ppe_vppctrl_input Success in
>>> delete in uecontext error
>>> 5 an_ppe_router_input packets
>>> received in rtr error
>>> 5 an_ppe_router_input packets dropped to host.
>>> no tacptcp session found error
>>> 14 an-ppe-isakmp4-output Total IKEV4 packets
>>> dispatched to network error
>>> 14 an-ppe-isakmpmgr-input packets processed by
>>> isakmpmgr input plugin error
>>> 14 an-ppe-isakmpmgr-input Received
>>> IKE packets error
>>> 14 an-ppe-isakmpmgr-input Successfully sent ike
>>> message to Session Manager error
>>> 5 an-ppe-isakmpmgr-input Received ike
>>> exchange AUTH packet error
>>> 4 an-ppe-isakmpmgr-input Received ike exchange
>>> CREATE CHILD SA packet error
>>> 1 an-ppe-isakmpmgr-input Received ike
>>> exchange SA INIT packet error
>>> 4 an-ppe-isakmpmgr-input Received ike
>>> exchange INFORMATIONAL packet error
>>> 3 arp-reply ARP
>>> replies sent error
>>> 14 ip4-udp-lookup No
>>> error error
>>> 5 esp4-decrypt-tun ESP pkts
>>> received error
>>> 5 ipsec4-tun-input good
>>> packets received error
>>> 1 ip4-input ip4
>>> ttl <= 1 error
>>> 1 ip4-icmp-error hop limit
>>> exceeded response sent error
>>> 27628 ethernet-input
>>> unknown vlan error
>>> vpp#
>>> vpp#
>>>
>>>
>>>
>>>
>>> On Wed, Mar 16, 2022 at 6:52 AM Florin Coras <[email protected]>
>>> wrote:
>>>
>>>> Hi Vijay,
>>>>
>>>> You won’t see the syn/syn-acks in traces because of the way they are
>>>> generated. Nonetheless, you can verify that the syns were received with
>>>> “show error” and check that the syn-acks were actually generated with a
>>>> pcap trace rxtx <output interface>
>>>>
>>>> Previously tracing worked because buffers were re-used and incoming
>>>> packets were directly sent to output. More recently we’ve started
>>>> dispatching syn-acks through the session layer in order to minimize size of
>>>> tx bursts per dispatch.
>>>>
>>>> Regards,
>>>> Florin
>>>>
>>>> On Mar 15, 2022, at 6:08 PM, Vijay Kumar <[email protected]> wrote:
>>>>
>>>> Hi Florin,
>>>>
>>>> Thanks for the quick response.
>>>>
>>>> It means it is expected to see the SYN pkts getting dropped after it is
>>>> terminated but I was not able to see any SYN-ACK going out of VPP.
>>>>
>>>> In both, "show trace" cmd output and also the pcap taken at dspatch
>>>> graph node level, I was not able to see the SYN-ACK going out. The route
>>>> to the SYN-ACK destination (which is the original source that started SYN)
>>>> is also present in the ip fib output. The configuration is the same that
>>>> was working fine for me in vp 20.05.
>>>>
>>>> Is there anything that I can look at for the SYN-ACK not sending issue?
>>>>
>>>>
>>>>
>>>> On Wed, 16 Mar 2022, 00:40 Florin Coras, <[email protected]>
>>>> wrote:
>>>>
>>>>> Hi Vijay,
>>>>>
>>>>> I see an_ppe_router_input forwards syns to tcp-input and those packets
>>>>> are delivered to tcp-listen, i.e., you’ve created a listener that’s
>>>>> matched
>>>>> by the incoming traffic.
>>>>>
>>>>> The thing to keep in mind is that tcp terminates incoming flows and it
>>>>> does not reuse buffers. That is, the syn hits the listen node and a
>>>>> syn-ack
>>>>> is programmed for this new connection that still needs to complete the
>>>>> handshake. The original syn packet is discarded and therefore you see it
>>>>> as
>>>>> a drop.
>>>>>
>>>>> Regards,
>>>>> Florin
>>>>>
>>>>> On Mar 15, 2022, at 3:50 AM, Vijay Kumar <[email protected]>
>>>>> wrote:
>>>>>
>>>>> The is the output of show trace and show interface
>>>>> ============================================
>>>>>
>>>>> Packet 36
>>>>>
>>>>> 00:03:26:875694: dpdk-input
>>>>> VirtualFuncEthernet0/7/0 rx queue 0
>>>>> buffer 0x4c1b21: current data 0, length 138, buffer-pool 0,
>>>>> ref-count 1, totlen-nifb 0, trace handle 0x1000023
>>>>> ext-hdr-valid
>>>>> l4-cksum-computed l4-cksum-correct
>>>>> PKT MBUF: port 0, nb_segs 1, pkt_len 138
>>>>> buf_len 2176, data_len 138, ol_flags 0x182, data_off 128,
>>>>> phys_addr 0xc266c8c0
>>>>> packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
>>>>> rss 0xde0c21da fdir.hi 0x0 fdir.lo 0xde0c21da
>>>>> Packet Offload Flags
>>>>> PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
>>>>> PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
>>>>> PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
>>>>> Packet Types
>>>>> RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
>>>>> RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or
>>>>> without extension headers
>>>>> RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet
>>>>> IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556
>>>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
>>>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
>>>>> fragment id 0x0000, flags DONT_FRAGMENT
>>>>> 00:03:26:875702: ethernet-input
>>>>> frame: flags 0x3, hw-if-index 3, sw-if-index 3
>>>>> IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556
>>>>> 00:03:26:875708: ip4-input
>>>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
>>>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
>>>>> fragment id 0x0000, flags DONT_FRAGMENT
>>>>> 00:03:26:875714: ip4-lookup
>>>>> fib 2 dpo-idx 27 flow hash: 0x00000000
>>>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
>>>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
>>>>> fragment id 0x0000, flags DONT_FRAGMENT
>>>>> 00:03:26:875715: ip4-local
>>>>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
>>>>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn
>>>>> NON_ECN
>>>>> fragment id 0x0000, flags DONT_FRAGMENT
>>>>> 00:03:26:875717: ipsec4-tun-input
>>>>> IPSec: remote:20.20.99.215 spi:4213 (0x00001075) sa:1 tun:0 seq 1 sa
>>>>> -2029505104
>>>>> 00:03:26:875719: esp4-decrypt-tun
>>>>> esp: crypto aes-cbc-256 integrity sha1-96 pkt-seq 1 sa-seq 1
>>>>> sa-seq-hi 0
>>>>> 00:03:26:875801: ip4-input-no-checksum
>>>>> TCP: 44.44.44.44 -> 30.30.30.30
>>>>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
>>>>> fragment id 0x692e, flags DONT_FRAGMENT
>>>>> TCP: 45723 -> 1234
>>>>> seq. 0x119d35d0 ack 0x00000000
>>>>> flags 0x02 SYN, tcp header: 40 bytes
>>>>> window 64240, checksum 0xd6be
>>>>> 00:03:26:875803: ip4-lookup
>>>>> fib 2 dpo-idx 25 flow hash: 0x00000000
>>>>> TCP: 44.44.44.44 -> 30.30.30.30
>>>>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
>>>>> fragment id 0x692e, flags DONT_FRAGMENT
>>>>> TCP: 45723 -> 1234
>>>>> seq. 0x119d35d0 ack 0x00000000
>>>>> flags 0x02 SYN, tcp header: 40 bytes
>>>>> window 64240, checksum 0xd6be
>>>>> 00:03:26:875803: ip4-local
>>>>> TCP: 44.44.44.44 -> 30.30.30.30
>>>>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
>>>>> fragment id 0x692e, flags DONT_FRAGMENT
>>>>> TCP: 45723 -> 1234
>>>>> seq. 0x119d35d0 ack 0x00000000
>>>>> flags 0x02 SYN, tcp header: 40 bytes
>>>>> window 64240, checksum 0xd6be
>>>>> 00:03:26:875805: an_ppe_router_input
>>>>> AN_PPE_ROUTER_INPUT: sw_if_index 1, next index 3
>>>>>
>>>>> 00:03:26:875819: tcp4-input
>>>>> [0:0][T] :::0->:::0 state CLOSED
>>>>> TCP: 45723 -> 1234
>>>>> seq. 0x119d35d0 ack 0x00000000
>>>>> flags 0x02 SYN, tcp header: 40 bytes
>>>>> window 64240, checksum 0xd6be
>>>>> 00:03:26:875826: tcp4-listen
>>>>> 1234 -> 45723 (LISTEN)
>>>>>
>>>>>
>>>>> Interface details
>>>>> ====================
>>>>> vpp# show interface
>>>>> Name Idx State MTU (L3/IP4/IP6/MPLS)
>>>>> Counter Count
>>>>> VirtualFuncEthernet0/7/0 3 up 1500/0/0/0 rx
>>>>> packets 3922
>>>>> rx
>>>>> bytes 239610
>>>>> tx
>>>>> packets 17
>>>>> tx
>>>>> bytes 6704
>>>>>
>>>>> drops 3914
>>>>> VirtualFuncEthernet0/7/0.1557 13 up 1500/0/0/0
>>>>> VirtualFuncEthernet0/7/0.1556 14 up 1500/0/0/0 rx
>>>>> packets 22
>>>>> rx
>>>>> bytes 5130
>>>>> tx
>>>>> packets 17
>>>>> tx
>>>>> bytes 6704
>>>>>
>>>>> drops 14
>>>>>
>>>>> ip4 19
>>>>> VirtualFuncEthernet0/8/0 4 down 9000/0/0/0
>>>>> gre1 2 up 9000/0/0/0
>>>>> ipip0 1 up 9000/0/0/0 rx
>>>>> packets 5
>>>>> rx
>>>>> bytes 500
>>>>>
>>>>> ip4 5
>>>>> local0 0 down 0/0/0/0
>>>>> loop2 8 up 9000/0/0/0
>>>>> loop3 9 up 9000/0/0/0
>>>>> loop4 10 up 9000/0/0/0
>>>>> loop5 11 up 9000/0/0/0
>>>>> loop6 12 up 9000/0/0/0
>>>>> memif0/0 5 up 0/65535/0/0 rx
>>>>> packets 1036
>>>>> rx
>>>>> bytes 67766832
>>>>> tx
>>>>> packets 1049
>>>>> tx
>>>>> bytes 67822842
>>>>>
>>>>> drops 2
>>>>>
>>>>> ip4 1036
>>>>> memif128/0 6 up 0/65535/0/0
>>>>> memif192/0 7 up 0/65535/0/0
>>>>> memif210/0 15 up 0/65535/0/0
>>>>> memif210/1 16 up 0/65535/0/0
>>>>> vpp#
>>>>> vpp#
>>>>>
>>>>> On Tue, Mar 15, 2022 at 2:32 PM Vijay Kumar via lists.fd.io
>>>>> <[email protected]> wrote:
>>>>>
>>>>>> Hi experts,
>>>>>>
>>>>>> I recently integrated vpp stack 21.06 and am running my NAS TCP
>>>>>> protocol test cases. The pcap dispatch trace that I captured shows the
>>>>>> TCP
>>>>>> SYN packets are coming to tcp-input() and then to tcp-listen() but from
>>>>>> here pkts are dropped instead of going to tcp-output() graph nodes for
>>>>>> sending the SYN-ACK
>>>>>>
>>>>>> NOTE
>>>>>> =========
>>>>>> The session layer is enabled and the TCP listen IP/port is also
>>>>>> showing in the session verbose command.
>>>>>>
>>>>>> The same test case used to work fine in the VPP 20.05. Am I missing
>>>>>> anything?
>>>>>> Let me know if there is anything I need to look at.
>>>>>>
>>>>>> vpp# show session verbose
>>>>>> Connection State
>>>>>> Rx-f Tx-f
>>>>>> [0:0][T] 30.30.30.30:1234->0.0.0.0:0 LISTEN
>>>>>> 0 0
>>>>>> Thread 0: active sessions 1
>>>>>> Thread 1: no sessions
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21037): https://lists.fd.io/g/vpp-dev/message/21037
Mute This Topic: https://lists.fd.io/mt/89793823/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
