Hi,

I set pmd-cpu-mask on a server running its processors in
cluster-on-die mode. This means, that the actual cpu topology is shown
to the os as shown below.

The problem I have is that although OVS is allowed to use all available CPUs:

$ ovs-vsctl --no-wait get Open_vSwitch . other_config:pmd-cpu-mask
"0x1787586c4fa8a01c71bc"
(=python hex(int('111111111111111111111111')))

OVS only uses the CPUs that are located in the first NUMA Node where
the OS is running:
$ ovs-appctl dpif-netdev/pmd-stats-show | grep ^pmd
pmd thread numa_id 0 core_id 2:
pmd thread numa_id 0 core_id 12:
pmd thread numa_id 0 core_id 13:
pmd thread numa_id 0 core_id 14:

I restarted OVS multiple times, I tried to pin queues to specific cores:

ovs-vsctl set interface dpdk0 other_config:pmd-rxq-affinity="0:2,1:12"
ovs-vsctl set interface dpdk1 other_config:pmd-rxq-affinity="0:13,1:14"
ovs-vsctl set interface vif3 other_config:pmd-rxq-affinity="0:3,1:3"
ovs-vsctl set interface vif4 other_config:pmd-rxq-affinity="0:4,1:4"

but with the same result, cores in other numa nodes are not used:

/usr/local/var/log/openvswitch/ovs-vswitchd.log
2016-09-18T15:25:04.327Z|00080|dpif_netdev|INFO|Created 4 pmd threads
on numa node 0
2016-09-18T15:25:04.327Z|00081|dpif_netdev|WARN|There is no PMD thread
on core 3. Queue 0 on port 'vif3' will not be polled.
2016-09-18T15:25:04.327Z|00082|dpif_netdev|WARN|There is no PMD thread
on core 4. Queue 0 on port 'vif4' will not be polled.
2016-09-18T15:25:04.327Z|00083|dpif_netdev|WARN|There's no available
pmd thread on numa node 0
2016-09-18T15:25:04.327Z|00084|dpif_netdev|WARN|There's no available
pmd thread on numa node 0

The log output seems to indicate that only numa node 0 is used for
some reason? Can anyone confirm this?

Best regards,

Jeremias


ovs-vsctl show
    Bridge "br0"
        Controller "tcp:<ctrl>:6633"
            is_connected: true
        Port "vif0"
            Interface "vif0"
                type: dpdkvhostuser
                options: {n_rxq="2"}
        Port "dpdk1"
            Interface "dpdk1"
                type: dpdk
                options: {n_rxq="2"}
        Port "vif3"
            Interface "vif3"
                type: dpdkvhostuser
                options: {n_rxq="2"}
        Port "dpdk0"
            Interface "dpdk0"
                type: dpdk
                options: {n_rxq="2"}
        Port "vif1"
            Interface "vif1"
                type: dpdkvhostuser
                options: {n_rxq="2"}
        Port "br0"
            Interface "br0"
                type: internal
        Port "vif4"
            Interface "vif4"
                type: dpdkvhostuser
                options: {n_rxq="2"}
    ovs_version: "2.6.90"


OVS (last commit):
commit 75e2077e0c43224bcca92746b28b01a4936fc101
Author: Thadeu Lima de Souza Cascardo <casca...@redhat.com>
Date:   Fri Sep 16 15:52:48 2016 -0300


CPU topology:
lstopo -p

Machine (252GB total)
  Package P#0
    NUMANode P#0 (63GB) +  L3 (15MB)
        L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#0
        L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#1
        L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#4 + PU P#2
        L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#12
        L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#13
        L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#5 + PU P#14
    NUMANode P#1 (63GB) + L3 (15MB)
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#8 + PU P#3
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#10 + PU P#4
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#12 + PU P#5
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#9 + PU P#15
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#11 + PU P#16
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#13 + PU P#17
  Package P#1
    NUMANode P#2 (63GB) + L3 (15MB)
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#6
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#7
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#4 + PU P#8
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#18
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#19
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#5 + PU P#20
    NUMANode P#3 (63GB) + L3 (15MB)
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#8 + PU P#9
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#10 + PU P#10
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#12 + PU P#11
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#9 + PU P#21
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#11 + PU P#22
      L2 (256KB) + L1d (32KB) + L1i (32KB) + Core P#13 + PU P#23

Other info:

$ uname -a
Linux nfvi1 4.4.0-34-lowlatency #53-Ubuntu SMP PREEMPT Wed Jul 27
19:23:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.4.0-34-lowlatency root=UUID=<whatever> ro
default_hugepagesz=1GB hugepagesz=1G hugepages=100 isolcpus=2-23
nohz_full=2-23 rcu_nocbs=2-23 apparmor=0
_______________________________________________
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Reply via email to