Hi Sejun,
I’ve never tried spdk, so I can’t know for sure where the issue lies but
possible sources of problems could be:
- mlx5 driver in combination with vpp. Have you tried vpp’s native rdma based
driver [1]?
- TSO is not enabled by default for tcp in vpp. To enable it, you’d need to
change vpp startup.conf to
- add a tcp { tso } stanza [2]
- under the dpdk stanza add "enable-tcp-udp-checksum” and under your
device’s config add “tso on”, in addition to the number of tx and rx
descriptors. I’ve only tested this with xl710s, so I don’t know if it works
fine with mlx5.
- default mtu for tcp is 1460. If you want to use jumbo frames change
startup.conf and under tcp stanza add “mtu 9000”. Never tested this in
combination with tso, so you may not want to mix them.
- what sizes is SPDK using for rx/tx fifos when using vpp’s host stack? Given
that writes/reads are as large as 128k, the fifos should probably be pretty
large, say 4-6MB.
- ensure that the cores spdk is using are not overlapping vpp’s worker(s) and
that the interface and the workers (vpp’s and spdk’s) are on the same numa [3].
Finally, what sort of throughput are you seeing and with how many workers? What
does the vector rate look like in vpp, i.e., execute "sh run” in the cli and
check dpdk-input and tcp-output nodes?
For reference, 1 connection with xl710s and 4MB fifos between 2 directly
connected hosts should reach over 37Gbps [4].
Regards,
Florin
[1] https://git.fd.io/vpp/tree/src/plugins/rdma/rdma_doc.md
[2] You may also want to switch to cubic as a congestion control algorithm by
adding to the tcp stanza {cc-algo cubic}.
[3] To check the numa for your interface do “sh hardware”
[4] https://wiki.fd.io/view/VPP/HostStack/LDP/iperf
> On Feb 14, 2020, at 11:07 PM, sejun.kwon via Lists.Fd.Io
> <[email protected]> wrote:
>
> Hello, I'm working on SPDK library + VPP, because some report said that VPP
> reduces the overhead of network. When I test with VPP (with mlx5 poll mode
> driver mtu 9000) and null device with spdk, 4k performance with VPP is much
> better than the default(kTCP). But, 128k write performance with VPP is 30
> percent lower than the one with kTCP. Is there anyone know why 128k write
> performance with VPP is not good as kernel ? I increase Multi thread or
> num-rx-desc and there is improvment, but it still lower than kernel. Is there
> also build option related to performance? Thanks in advance.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#15409): https://lists.fd.io/g/vpp-dev/message/15409
Mute This Topic: https://lists.fd.io/mt/71294801/21656
Mute #vpp: https://lists.fd.io/mk?hashtag=vpp&subid=1480452
Mute #mellanox: https://lists.fd.io/mk?hashtag=mellanox&subid=1480452
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
