If I do not manually assign cores to nic, and let vpp assign cores, vpp
only assign one core per nic queue. If I also assign number of queues to
10, all cores are used. Otherwise, only core 4 and 6 are assigned to each
nic, which has only one queue.
Probably, there is some smart logic adaptively assigning cores to nics?
Anyway, a single core per nic is good enough to us for now.
Without specifying number of rx queues per nic, only core 4 and 6 are
assigned.
vpp# sh threads
ID Name Type LWP Sched Policy (Priority)
lcore Core Socket State
0 vpp_main 16886 other (0) 2
0 0
1 vpp_wk_0 workers 16888 other (0) 4
1 0
2 vpp_wk_1 workers 16889 other (0) 6
4 0
3 vpp_wk_2 workers 16890 other (0) 8
2 0
4 vpp_wk_3 workers 16891 other (0) 10
3 0
5 vpp_wk_4 workers 16892 other (0) 12
8 0
6 vpp_wk_5 workers 16893 other (0) 14
13 0
7 vpp_wk_6 workers 16894 other (0) 16
9 0
8 vpp_wk_7 workers 16895 other (0) 18
12 0
9 vpp_wk_8 workers 16896 other (0) 20
10 0
10 vpp_wk_9 workers 16897 other (0) 22
11 0
vpp# sh interface rx-placement
Thread 1 (vpp_wk_0):
node dpdk-input:
eth1 queue 0 (polling)
Thread 2 (vpp_wk_1):
node dpdk-input:
eth0 queue 0 (polling)
vpp#
cpu {
main-core 2
corelist-workers 4,6,8,10,12,14,16,18,20,22
}
dpdk {
socket-mem 2048,0
log-level debug
no-tx-checksum-offload
dev default{
num-tx-desc 512
num-rx-desc 512
}
dev 0000:1a:00.0 {
name eth0
}
dev 0000:19:00.1 {
name eth1
}
# Use aesni mb lib.
vdev crypto_aesni_mb0,socket_id=0
no-multi-seg
}
When specifying number of queues per nic, all cores are used.
vpp# sh int rx-placement
Thread 1 (vpp_wk_0):
node dpdk-input:
eth1 queue 0 (polling)
eth0 queue 0 (polling)
Thread 2 (vpp_wk_1):
node dpdk-input:
eth1 queue 1 (polling)
eth0 queue 1 (polling)
Thread 3 (vpp_wk_2):
node dpdk-input:
eth1 queue 2 (polling)
eth0 queue 2 (polling)
Thread 4 (vpp_wk_3):
node dpdk-input:
eth1 queue 3 (polling)
eth0 queue 3 (polling)
Thread 5 (vpp_wk_4):
node dpdk-input:
eth1 queue 4 (polling)
eth0 queue 4 (polling)
Thread 6 (vpp_wk_5):
node dpdk-input:
eth1 queue 5 (polling)
eth0 queue 5 (polling)
Thread 7 (vpp_wk_6):
node dpdk-input:
eth1 queue 6 (polling)
eth0 queue 6 (polling)
Thread 8 (vpp_wk_7):
node dpdk-input:
eth1 queue 7 (polling)
eth0 queue 7 (polling)
Thread 9 (vpp_wk_8):
node dpdk-input:
eth1 queue 8 (polling)
eth0 queue 8 (polling)
Thread 10 (vpp_wk_9):
node dpdk-input:
eth1 queue 9 (polling)
eth0 queue 9 (polling)
vpp# sh threads
ID Name Type LWP Sched Policy (Priority)
lcore Core Socket State
0 vpp_main 16905 other (0) 2
0 0
1 vpp_wk_0 workers 16907 other (0) 4
1 0
2 vpp_wk_1 workers 16908 other (0) 6
4 0
3 vpp_wk_2 workers 16909 other (0) 8
2 0
4 vpp_wk_3 workers 16910 other (0) 10
3 0
5 vpp_wk_4 workers 16911 other (0) 12
8 0
6 vpp_wk_5 workers 16912 other (0) 14
13 0
7 vpp_wk_6 workers 16913 other (0) 16
9 0
8 vpp_wk_7 workers 16914 other (0) 18
12 0
9 vpp_wk_8 workers 16915 other (0) 20
10 0
10 vpp_wk_9 workers 16916 other (0) 22
11 0
vpp#
cpu {
main-core 2
corelist-workers 4,6,8,10,12,14,16,18,20,22
}
dpdk {
socket-mem 2048,0
log-level debug
no-tx-checksum-offload
dev default{
num-tx-desc 512
num-rx-desc 512
num-rx-queues 10
}
dev 0000:1a:00.0 {
name eth0
}
dev 0000:19:00.1 {
name eth1
}
# Use aesni mb lib.
vdev crypto_aesni_mb0,socket_id=0
no-multi-seg
}
On Fri, Nov 8, 2019 at 12:54 AM Benoit Ganne (bganne) via Lists.Fd.Io
<[email protected]> wrote:
> Hi Chuan,
>
> > The weird thing is that when I reduced the number of workers
> > everything worked fine. I did send 8.5Gbps udp/tcp traffic over the two
> > machines. I also saw encryption/decryption happening. How could this be
> > possible without crypto engine?
>
> Hmm that should not have happened :)
> Frankly I have no idea, but maybe there was other difference in
> configuration?
>
> > I installed vpp-plugin-core. No more crash was seen.
> > However, when I reserved 10 cores for 2 nics. Some cores are not polling
> > shown by htop.
> [...]
> > Any clue why? Or, there is probably more misconfig here? Or bug?
>
> The config looks sane so it might be a bug in the way we parse dpdk
> workers conf. If instead of manually assigning cores to NIC you let VPP
> assigns them is it better?, eg.
>
> dpdk {
> socket-mem 2048,0
> log-level debug
> no-tx-checksum-offload
> dev default{
> num-tx-desc 512
> num-rx-desc 512
> num-rx-queues 10
> }
> dev 0000:1a:00.0 {
> name eth0
> }
> dev 0000:19:00.1 {
> name eth1
> }
> }
>
> Thx
> ben
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#14542): https://lists.fd.io/g/vpp-dev/message/14542
> Mute This Topic: https://lists.fd.io/mt/41334952/1991531
> Group Owner: [email protected]
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]]
> -=-=-=-=-=-=-=-=-=-=-=-
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14566): https://lists.fd.io/g/vpp-dev/message/14566
Mute This Topic: https://lists.fd.io/mt/41334952/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
