The tx-error usually indicates the NIC is too slow to pick up the packets for
tx, but I suspect you have a configuration issue somewhere because you should
be able to route 8Mpps of IPv6 with 1 or 2 workers.
Can you give us the output of (while traffic is going through):
- show pci
- show threads
- show int rx
- show hardware
- show buffers
- clear run / show run
Best
ben
> -----Original Message-----
> From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of
> jacicson1...@163.com
> Sent: vendredi 10 juillet 2020 13:46
> To: vpp-dev@lists.fd.io
> Subject: [vpp-dev] #vpp How can I get rid of tx-errors with 100G NICs ?
> #dpdk
>
>
> I tested single IPv6 route performance with two 100G NICs.
> The test tool is Trex which can send packets 100Gb/s easily.
> But VPP(1908.1) has a lot of tx-errors while forwarding the packets.
>
> 20 threads, 20 rx queues, 20 tx queues, 60G Hugepages.
> When T-rex send 100Gb/s, VPP droped nearly 10% packets.
>
> How can I get rid of the tx-error? Is it possible to improve the
> performance?
> When I set 30 workers or 10 workers with same tx/rx queues, the tx-error
> is not changed.
>
> Information
>
> Trex's result:
> Send 8.2Mpps, Recv 7.33Mpps
>
> Global Statistitcs
>
> connection : localhost, Port 4501 total_tx_L2 : 98.71
> Gb/sec
> version : STL @ v2.81 total_tx_L1 : 100.02
> Gb/sec
> cpu_util. : 5.67% @ 14 cores (14 per dual port) total_rx : 88.25
> Gb/sec
> rx_cpu_util. : 0.0% / 0 pkt/sec total_pps : 8.2
> Mpkt/sec
> async_util. : 0.16% / 1.32 KB/sec drop_rate : 10.46
> Gb/sec
> total_cps. : 0 cps/sec queue_full : 0 pkts
>
> Port Statistics
>
> port | 0 | 1 | total
> -----------+-------------------+-------------------+------------------
> owner | root | root |
> link | UP | UP |
> state | TRANSMITTING | IDLE |
> speed | 100 Gb/s | 100 Gb/s |
> CPU util. | 5.67% | 0.0% |
> -- | | |
> Tx bps L2 | 98.71 Gbps | 0 bps | 98.71 Gbps
> Tx bps L1 | 100.02 Gbps | 0 bps | 100.02 Gbps
> Tx pps | 8.2 Mpps | 0 pps | 8.2 Mpps
> Line Util. | 100.02 % | 0 % |
> --- | | |
> Rx bps | 0 bps | 88.25 Gbps | 88.25 Gbps
> Rx pps | 0 pps | 7.33 Mpps | 7.33 Mpps
> ---- | | |
> opackets | 223589643 | 0 | 223589643
> ipackets | 2 | 199741988 | 199741990
> obytes | 336278835072 | 0 | 336278835072
> ibytes | 180 | 300411956124 | 300411956304
> tx-pkts | 223.59 Mpkts | 0 pkts | 223.59 Mpkts
> rx-pkts | 2 pkts | 199.74 Mpkts | 199.74 Mpkts
> tx-bytes | 336.28 GB | 0 B | 336.28 GB
> rx-bytes | 180 B | 300.41 GB | 30
>
>
> vpp# show int
> Name Idx State MTU (L3/IP4/IP6/MPLS)
> Counter Count
> HundredGigabitEthernet82/0/0 1 up 9000/0/0/0 rx
> packets 5038457978
> rx
> bytes 7557686967000
> tx
> packets 8
> tx
> bytes 752
> ip6
> 5038458089
> rx-
> miss 29522
> HundredGigabitEthernet83/0/0 2 up 9000/0/0/0 tx
> packets 5038458485
> tx
> bytes 7557687717666
> tx-
> error 526772148
> local0 0 down 0/0/0/0
>
> vpp# show err
> Count Node Reason
> 14 ip6-icmp-input router advertisements
> sent
> 32038454 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 32036988 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 32038478 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 32027733 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 32038748 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 32042218 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 32038604 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 32029906 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 32038974 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 32041972 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 32038119 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 32032684 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 21625413 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 21627658 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 21627926 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 21610610 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 21618128 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 21614224 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 21615517 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
> 21617616 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
> failure)
>
> vpp# show int addr
> HundredGigabitEthernet82/0/0 (up):
> L3 2d01:1111::1/64
> HundredGigabitEthernet83/0/0 (up):
> L3 2d02:1111::1/64
> local0 (dn):
> vpp#
> vpp# show ip6 fib
> ipv6-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ]
> locks:[src:plugin-hi:1, src:recursive-resolution:1, src:default-route:1, ]
> ::/0
> unicast-ip6-chain
> [@0]: dpo-load-balance: [proto:ip6 index:6 buckets:1 uRPF:15
> to:[6015241112:8938648292432]]
> [0] [@19]: dpo-load-balance: [proto:ip6 index:19 buckets:1 uRPF:14
> to:[0:0] via:[6015241112:8938648292432]]
> [0] [@5]: ipv6 via 2d02:1111::100 HundredGigabitEthernet83/0/0:
> mtu:9000 0c42a150d9860c42a150d98286dd
>
> Hugepages:
> cat /proc/meminfo | grep Hu
> AnonHugePages: 114688 kB
> HugePages_Total: 60
> HugePages_Free: 54
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 1048576 kB
>
> startup.conf
> unix {
> nodaemon
> log /var/log/vpp/vpp.log
> full-coredump
> cli-listen /run/vpp/cli.sock
> gid vpp
> }
>
> api-trace {
> ## This stanza controls binary API tracing. Unless there is a very strong
> reason,
> ## please leave this feature enabled.
> on
> ## Additional parameters:
> ##
> ## To set the number of binary API trace records in the circular buffer,
> configure nitems
> ##
> ## nitems <nnn>
> ##
> ## To save the api message table decode tables, configure a filename.
> Results in /tmp/<filename>
> ## Very handy for understanding api message changes between versions,
> identifying missing
> ## plugins, and so forth.
> ##
> ## save-api-table <filename>
> }
>
> api-segment {
> gid vpp
> }
>
> socksvr {
> default
> }
>
> cpu {
> ## In the VPP there is one main thread and optionally the user can
> create worker(s)
> ## The main thread and worker thread(s) can be pinned to CPU
> core(s) manually or automatically
>
> ## Manual pinning of thread(s) to CPU core(s)
>
> ## Set logical CPU core where main thread runs, if main core is
> not set
> ## VPP will use core 1 if available
> #main-core 3
>
> ## Set logical CPU core(s) where worker threads are running
> #corelist-workers 5,7,9,11,13,15,17,19,21,23
>
> ## Automatic pinning of thread(s) to CPU core(s)
>
> ## Sets number of CPU core(s) to be skipped (1 ... N-1)
> ## Skipped CPU core(s) are not used for pinning main thread and
> working thread(s).
> ## The main thread is automatically pinned to the first available
> CPU core and worker(s)
> ## are pinned to next free CPU core(s) after core assigned to main
> thread
> skip-cores 2
>
> ## Specify a number of workers to be created
> ## Workers are pinned to N consecutive CPU cores while skipping
> "skip-cores" CPU core(s)
> ## and main thread's CPU core
> workers 20
>
> ## Set scheduling policy and priority of main and worker threads
>
> ## Scheduling policy options are: other (SCHED_OTHER), batch
> (SCHED_BATCH)
> ## idle (SCHED_IDLE), fifo (SCHED_FIFO), rr (SCHED_RR)
> # scheduler-policy fifo
>
> ## Scheduling priority is used only for "real-time policies (fifo
> and rr),
> ## and has to be in the range of priorities supported for a
> particular policy
> # scheduler-priority 50
> }
>
> buffers {
> ## Increase number of buffers allocated, needed only in scenarios
> with
> ## large number of interfaces and worker threads. Value is per
> numa node.
> ## Default is 16384 (8192 if running unpriviledged)
> buffers-per-numa 512000
>
> ## Size of buffer data area
> ## Default is 2048
> # default data-size 2048
> }
>
> dpdk {
> ## Change default settings for all interfaces
> dev default {
> ## Number of receive queues, enables RSS
> ## Default is 1
> num-rx-queues 20
>
> ## Number of transmit queues, Default is equal
> ## to number of worker threads or 1 if no workers treads
> num-tx-queues 20
>
> ## Number of descriptors in transmit and receive rings
> ## increasing or reducing number can impact performance
> ## Default is 1024 for both rx and tx
> #num-rx-desc 2048
> #num-tx-desc 2048
>
> ## VLAN strip offload mode for interface
> ## Default is off
> # vlan-strip-offload on
> }
>
> ## Whitelist specific interface by specifying PCI address
> dev 0000:82:00.0
> dev 0000:83:00.0
> #vdev
> eth_bond0,mode=2,slave=0000:02:00.0,slave=0000:03:00.0,slave=0000:81:00.0,
> xmit_policy=l34
> #vdev
> eth_bond1,mode=2,slave=0000:02:00.1,slave=0000:03:00.1,slave=0000:81:00.1,
> xmit_policy=l34
>
> ## Blacklist specific device type by specifying PCI vendor:device
> ## Whitelist entries take precedence
> # blacklist 8086:10fb
>
> ## Set interface name
> # dev 0000:02:00.1 {
> # name eth0
> # }
>
> ## Whitelist specific interface by specifying PCI address and in
> ## addition specify custom parameters for this interface
> # dev 0000:02:00.1 {
> # num-rx-queues 2
> # }
>
> ## Change UIO driver used by VPP, Options are: igb_uio, vfio-pci,
> ## uio_pci_generic or auto (default)
> # uio-driver vfio-pci
>
> ## Disable multi-segment buffers, improves performance but
> ## disables Jumbo MTU support
> # no-multi-seg
>
> ## Change hugepages allocation per-socket, needed only if there is
> need for
> ## larger number of mbufs. Default is 256M on each detected CPU
> socket
> socket-mem 2048,2048
>
> ## Disables UDP / TCP TX checksum offload. Typically needed for
> use
> ## faster vector PMDs (together with no-multi-seg)
> # no-tx-checksum-offload
> }
>
> NICs:
>
> Mellanox ConnectX-5 MCX555A-ECAT single port. x 2
>
> [root@localhost v2.81]# ./dpdk_nic_bind.py -s
>
> Network devices using DPDK-compatible driver
> ============================================
> 0000:82:00.0 'MT27800 Family [ConnectX-5]' drv=mlx5_core
> unused=uio_pci_generic
> 0000:83:00.0 'MT27800 Family [ConnectX-5]' drv=mlx5_core
> unused=uio_pci_generic
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16930): https://lists.fd.io/g/vpp-dev/message/16930
Mute This Topic: https://lists.fd.io/mt/75416173/21656
Mute #vpp: https://lists.fd.io/g/fdio+vpp-dev/mutehashtag/vpp
Mute #dpdk: https://lists.fd.io/g/fdio+vpp-dev/mutehashtag/dpdk
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
