Don't use tapcli code, it is outdated and I just submitted patch to deprecate it (as agreed few months ago on the community call).
https://gerrit.fd.io/r/17073 <https://gerrit.fd.io/r/17073> tapv2 (create interface tap) should be able to run on worker, but don't expect magic, it's jus tap.... > On 25 Jan 2019, at 08:14, ranadip....@gmail.com wrote: > > Hello, > > I am trying to use vpp with 4 worker threads. > Client is sending packet to server through vpp. This packet comes to vpp, and > gets picked up from a VF. VPP sends this packet to the another user-space > application over a tapcli interface. > The user-space application processes the packet, and then sends it back to > vpp. Vpp forwards the packet to the server. > > I am using vpp with 4 worker threads. > > Without the traffic, four worker threads are all showing ~100% cpu usage > (which is expected). However, with traffic, the vpp main thread is also > showing ~100% cpu usage. I do not want vpp main thread to reach 100% cpu. > With the limited knowledge that I have on vpp, it seems like main thread is > doing lots of processing in the tapcli-rx node. Is there a way to keep vpp > main relatively free? Is tap interface the correct choice to communicate > between vpp and another user-space application? If not, what should be used? > > I have tried this on both vpp 18.07 and 18.10. > > vpp# show run > Thread 0 vpp_main (lcore 2) > Time 916.9, average vectors/node 40.16, last 128 main loops 0.00 per node 0.00 > vector rates in 1.3178e5, out 1.3178e5, drop 1.3087e-2, punt 0.0000e0 > Name State Calls Vectors > Suspends Clocks Vectors/Call > VirtualFunctionEthernet3/10/1- active 3008704 120840447 > 0 1.36e1 40.16 > VirtualFunctionEthernet3/10/1- active 3008704 120840447 > 0 1.02e2 40.16 > acl-plugin-fa-cleaner-process event wait 0 0 > 1 1.85e4 0.00 > admin-up-down-process event wait 0 0 > 1 1.20e3 0.00 > api-rx-from-ring any wait 0 0 > 50 7.80e4 0.00 > arp-input active 1 1 > 0 2.39e6 1.00 > avf-process event wait 0 0 > 1 2.14e3 0.00 > bfd-process event wait 0 0 > 1 1.84e3 0.00 > bond-process event wait 0 0 > 1 1.44e3 0.00 > cdp-process any wait 0 0 > 1 1.38e7 0.00 > dhcp-client-process any wait 0 0 > 10 1.02e4 0.00 > dhcp6-client-cp-process any wait 0 0 > 1 1.07e3 0.00 > dhcp6-pd-client-cp-process any wait 0 0 > 1 9.28e2 0.00 > dhcp6-pd-reply-publisher-proce event wait 0 0 > 1 8.88e2 0.00 > dhcp6-reply-publisher-process event wait 0 0 > 1 1.02e3 0.00 > dns-resolver-process any wait 0 0 > 1 2.18e3 0.00 > dpdk-ipsec-process done 1 0 > 0 1.29e5 0.00 > dpdk-process any wait 0 0 > 306 5.73e4 0.00 > error-drop active 11 12 > 0 5.73e3 1.09 > ethernet-input active 3008710 120840454 > 0 3.78e1 40.16 > fib-walk any wait 0 0 > 459 3.03e3 0.00 > flow-report-process any wait 0 0 > 1 6.92e2 0.00 > flowprobe-timer-process any wait 0 0 > 1 3.09e3 0.00 > igmp-timer-process event wait 0 0 > 1 2.18e3 0.00 > ikev2-manager-process any wait 0 0 > 916 2.46e3 0.00 > ioam-export-process any wait 0 0 > 1 1.10e3 0.00 > ip-neighbor-scan-process any wait 0 0 > 16 7.03e3 0.00 > ip-route-resolver-process any wait 0 0 > 10 8.13e3 0.00 > ip4-glean active 2 2 > 0 9.34e3 1.00 > ip4-input active 3008704 120840447 > 0 4.03e1 40.16 > ip4-lookup active 3008704 120840447 > 0 3.67e1 40.16 > ip4-reassembly-expire-walk any wait 0 0 > 92 3.96e3 0.00 > ip4-rewrite active 3008702 120840445 > 0 3.08e1 40.16 > ip6-icmp-neighbor-discovery-ev any wait 0 0 > 916 2.00e3 0.00 > ip6-input active 5 6 > 0 1.17e3 1.20 > ip6-not-enabled active 5 6 > 0 5.45e2 1.20 > ip6-reassembly-expire-walk any wait 0 0 > 92 4.30e3 0.00 > l2fib-mac-age-scanner-process event wait 0 0 > 1 1.78e3 0.00 > lacp-process event wait 0 0 > 1 1.41e7 0.00 > lisp-retry-service any wait 0 0 > 459 4.11e3 0.00 > lldp-process event wait 0 0 > 1 1.31e7 0.00 > memif-process event wait 0 0 > 1 4.28e3 0.00 > nat-det-expire-walk done 1 0 > 0 1.23e3 0.00 > nat64-expire-walk event wait 0 0 > 1 1.46e4 0.00 > rd-cp-process any wait 0 0 > 1 9.16e2 0.00 > send-dhcp6-client-message-proc any wait 0 0 > 1 2.88e3 0.00 > send-dhcp6-pd-client-message-p any wait 0 0 > 1 1.02e3 0.00 > send-garp-na-process event wait 0 0 > 1 2.88e4 0.00 > send-rs-process any wait 0 0 > 1 1.24e3 0.00 > startup-config-process done 1 0 > 1 3.75e3 0.00 > tapcli-rx interrupt wa 5382120 120840506 > 0 2.42e3 22.45 > udp-ping-process any wait 0 0 > 1 5.70e3 0.00 > unix-cli-local:6 active 6 0 > 757 3.46e15 0.00 > unix-epoll-input polling 16312904 0 > 0 6.94e4 0.00 > vhost-user-process any wait 0 0 > 1 1.32e4 0.00 > vhost-user-send-interrupt-proc any wait 0 0 > 1 1.03e3 0.00 > vpe-link-state-process event wait 0 0 > 3 4.95e3 0.00 > vpe-oam-process any wait 0 0 > 450 2.86e3 0.00 > vxlan-gpe-ioam-export-process any wait 0 0 > 1 8.76e2 0.00 > wildcard-ip4-arp-publisher-pro event wait 0 0 > 1 1.09e3 0.00 > --------------- > Thread 1 vpp_wk_0 (lcore 3) > Time 916.9, average vectors/node 1.43, last 128 main loops 0.00 per node 0.00 > vector rates in 3.2809e4, out 3.2809e4, drop 5.6599e-1, punt 0.0000e0 > Name State Calls Vectors > Suspends Clocks Vectors/Call > arp-input active 518 518 > 0 7.74e3 1.00 > dpdk-input polling 3205687180 30084768 > 0 4.31e4 0.00 > error-drop active 519 519 > 0 1.28e3 1.00 > ethernet-input active 518 518 > 0 1.73e3 1.00 > ip4-glean active 1 1 > 0 6.62e4 1.00 > ip4-input-no-checksum active 20966739 30084250 > 0 3.64e2 1.43 > ip4-lookup active 20966739 30084250 > 0 2.46e2 1.43 > ip4-rewrite active 20966738 30084249 > 0 1.93e2 1.43 > tapcli-0-output active 20966739 30084250 > 0 2.03e2 1.43 > tapcli-0-tx active 20966739 30084250 > 0 1.29e4 1.43 > --------------- > Thread 2 vpp_wk_1 (lcore 4) > Time 916.9, average vectors/node 1.45, last 128 main loops 0.00 per node 0.00 > vector rates in 3.2819e4, out 3.2819e4, drop 0.0000e0, punt 0.0000e0 > Name State Calls Vectors > Suspends Clocks Vectors/Call > dpdk-input polling 3062149112 30094113 > 0 4.24e4 0.00 > ip4-input-no-checksum active 20687846 30094113 > 0 3.67e2 1.45 > ip4-lookup active 20687846 30094113 > 0 2.59e2 1.45 > ip4-rewrite active 20687846 30094113 > 0 2.03e2 1.45 > tapcli-0-output active 20687846 30094113 > 0 2.11e2 1.45 > tapcli-0-tx active 20687846 30094113 > 0 1.31e4 1.45 > --------------- > Thread 3 vpp_wk_2 (lcore 5) > Time 916.9, average vectors/node 1.46, last 128 main loops 0.00 per node 0.00 > vector rates in 3.2774e4, out 3.2774e4, drop 2.4428e-1, punt 0.0000e0 > Name State Calls Vectors > Suspends Clocks Vectors/Call > dpdk-input polling 2997474079 30052436 > 0 4.33e4 .01 > error-drop active 224 224 > 0 1.57e3 1.00 > ip4-drop active 224 224 > 0 8.69e2 1.00 > ip4-input-no-checksum active 20609931 30052436 > 0 3.74e2 1.46 > ip4-lookup active 20609931 30052436 > 0 2.64e2 1.46 > ip4-rewrite active 20609737 30052212 > 0 2.01e2 1.46 > tapcli-0-output active 20609737 30052212 > 0 2.15e2 1.46 > tapcli-0-tx active 20609737 30052212 > 0 1.31e4 1.46 > --------------- > Thread 4 vpp_wk_3 (lcore 6) > Time 916.9, average vectors/node 1.45, last 128 main loops 0.00 per node 0.00 > vector rates in 3.2811e4, out 3.2811e4, drop 0.0000e0, punt 0.0000e0 > Name State Calls Vectors > Suspends Clocks Vectors/Call > dpdk-input polling 3135905643 30086296 > 0 4.30e4 0.00 > ip4-input-no-checksum active 20714610 30086296 > 0 3.74e2 1.45 > ip4-lookup active 20714610 30086296 > 0 2.55e2 1.45 > ip4-rewrite active 20714610 30086296 > 0 1.97e2 1.45 > tapcli-0-output active 20714610 30086296 > 0 2.04e2 1.45 > tapcli-0-tx active 20714610 30086296 > 0 1.31e4 1.45 > > Here is one of the packets traced on vpp main thread. > > vpp# show trace > ------------------- Start of thread 0 vpp_main ------------------- > Packet 1 > > 00:11:26:605715: tapcli-rx > tapcli-0 > 00:11:26:605725: ethernet-input > IP4: c2:aa:32:4f:d8:1c -> 02:fe:d9:0f:5c:f4 > 00:11:26:605741: ip4-input > TCP: 10.0.1.1 -> 172.18.60.74 > tos 0x00, ttl 64, length 60, checksum 0x475f > fragment id 0x0000, flags DONT_FRAGMENT > TCP: 82 -> 60818 > seq. 0xcbbfa586 ack 0x36bb48c6 > flags 0x12 SYN ACK, tcp header: 40 bytes > window 28960, checksum 0xab2b > 00:11:26:605748: ip4-lookup > fib 0 dpo-idx 3 flow hash: 0x00000000 > TCP: 10.0.1.1 -> 172.18.60.74 > tos 0x00, ttl 64, length 60, checksum 0x475f > fragment id 0x0000, flags DONT_FRAGMENT > TCP: 82 -> 60818 > seq. 0xcbbfa586 ack 0x36bb48c6 > flags 0x12 SYN ACK, tcp header: 40 bytes > window 28960, checksum 0xab2b > 00:11:26:605754: ip4-rewrite > tx_sw_if_index 1 dpo-idx 3 : ipv4 via 172.18.60.74 > VirtualFunctionEthernet3/10/1: mtu:9000 ac1f6b17f379da78bd634ef70800 flow > hash: 0x00000000 > 00000000: ac1f6b17f379da78bd634ef708004500003c000040003f06485f0a000101ac12 > 00000020: 3c4a0052ed92cbbfa58636bb48c6a0127120ab2b0000020405b40402 > 00:11:26:605755: VirtualFunctionEthernet3/10/1-output > VirtualFunctionEthernet3/10/1 > IP4: da:78:bd:63:4e:f7 -> ac:1f:6b:17:f3:79 > TCP: 10.0.1.1 -> 172.18.60.74 > tos 0x00, ttl 63, length 60, checksum 0x485f > fragment id 0x0000, flags DONT_FRAGMENT > TCP: 82 -> 60818 > seq. 0xcbbfa586 ack 0x36bb48c6 > flags 0x12 SYN ACK, tcp header: 40 bytes > window 28960, checksum 0xab2b > 00:11:26:605758: VirtualFunctionEthernet3/10/1-tx > VirtualFunctionEthernet3/10/1 tx queue 0 > buffer 0x23e5a0: current data 0, length 74, free-list 0, clone-count 0, > totlen-nifb 0, trace 0x0 > l2-hdr-offset 0 l3-hdr-offset 14 > PKT MBUF: port 65535, nb_segs 1, pkt_len 74 > buf_len 2176, data_len 74, ol_flags 0x0, data_off 128, phys_addr > 0xe7996880 > packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 > rss 0x0 fdir.hi 0x0 fdir.lo 0x0 > IP4: da:78:bd:63:4e:f7 -> ac:1f:6b:17:f3:79 > TCP: 10.0.1.1 -> 172.18.60.74 > tos 0x00, ttl 63, length 60, checksum 0x485f > fragment id 0x0000, flags DONT_FRAGMENT > TCP: 82 -> 60818 > seq. 0xcbbfa586 ack 0x36bb48c6 > flags 0x12 SYN ACK, tcp header: 40 bytes > window 28960, checksum 0xab2b > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#11999): https://lists.fd.io/g/vpp-dev/message/11999 > Mute This Topic: https://lists.fd.io/mt/29534956/675642 > Mute #vpp: https://lists.fd.io/mk?hashtag=vpp&subid=1480514 > Group Owner: vpp-dev+ow...@lists.fd.io > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [dmar...@me.com] > -=-=-=-=-=-=-=-=-=-=-=- -- Damjan
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12000): https://lists.fd.io/g/vpp-dev/message/12000 Mute This Topic: https://lists.fd.io/mt/29534956/21656 Mute #vpp: https://lists.fd.io/mk?hashtag=vpp&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
