On Wed, Oct 26, 2016 at 06:08:24PM +0800, Bo Sun wrote: > Hi list, > > I'm using OVS 2.6.0 and DPDK 16.07 on CentOS 7.2, following the > instructions on ovs/INSTALL.DPDK.rst. > > When I start ovs-vswitchd, it can't function properly. > > I got warnings like "ovs|00014|timeval|WARN|Unreasonably long 6567ms poll > interval (232ms user, 5958ms system)" and many others. I searched on the > Internet, and found a similar situation which is fixed by upgrading kernel. > It seems that ovs-vswitchd hangs. So I also upgraded kernel, but it didn't > solve the problem. > > The logs of DPDK seems strange. I'm using an Intel 82599 10Gb Ethernet > Adapter, and bind it to vfio-pci kernel module. I also mount hugepages. I > can't figure out why it goes wrong. > > I'd appreciate any suggestions and help on this issue. > > Here are logs of ovs: > > Oct 26 05:27:41 localhost.localdomain ovsdb-server[13795]: > ovs|00001|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.6.0 > Oct 26 05:27:48 localhost.localdomain ovs-vsctl[13796]: > ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . > other_config:dpdk-init=true > Oct 26 05:27:51 localhost.localdomain ovsdb-server[13795]: > ovs|00002|memory|INFO|2300 kB peak resident set size after 10.0 seconds > Oct 26 05:27:51 localhost.localdomain ovsdb-server[13795]: > ovs|00003|memory|INFO|cells:16 monitors:0 > Oct 26 05:27:52 localhost.localdomain ovs-vsctl[13797]: > ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . > other_config:dpdk-socket-mem=256,256,256,256 > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13798]: > ovs|00001|socket_util|ERR|localhost: port must be specified > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13798]: > ovs|00002|vlog|INFO|opened log file /usr/local/var/log/ > openvswitch/ovs-vsctl.log > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13799]: > ovs|00003|ovs_numa|INFO|Discovered 20 CPU cores on NUMA node 0 > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13799]: > ovs|00004|ovs_numa|INFO|Discovered 20 CPU cores on NUMA node 1 > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13799]: > ovs|00005|ovs_numa|INFO|Discovered 20 CPU cores on NUMA node 2 > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13799]: > ovs|00006|ovs_numa|INFO|Discovered 20 CPU cores on NUMA node 3 > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13799]: > ovs|00007|ovs_numa|INFO|Discovered 4 NUMA nodes and 80 CPU cores > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13799]: > ovs|00008|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: > connecting... > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13799]: > ovs|00009|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: > connected > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13799]: > ovs|00010|dpdk|INFO|DPDK Enabled, initializing > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13799]: > ovs|00011|dpdk|INFO|No vhost-sock-dir provided - defaulting to > /usr/local/var/run/openvswitch > Oct 26 05:28:00 localhost.localdomain ovs-vswitchd[13799]: > ovs|00012|dpdk|INFO|EAL ARGS: ovs-vswitchd --socket-mem 256,256,256,256 -c > 0x00000001 > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: PMD: > bnxt_rte_pmd_init() called for (null) > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: PCI device > 0000:01:00.0 on NUMA socket 0 > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: probe > driver: 8086:1521 rte_igb_pmd > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: PCI device > 0000:01:00.1 on NUMA socket 0 > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: probe > driver: 8086:1521 rte_igb_pmd > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: PCI device > 0000:01:00.2 on NUMA socket 0 > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: probe > driver: 8086:1521 rte_igb_pmd > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: PCI device > 0000:01:00.3 on NUMA socket 0 > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: probe > driver: 8086:1521 rte_igb_pmd > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: PCI device > 0000:8b:00.0 on NUMA socket 2 > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: probe > driver: 8086:10fb rte_ixgbe_pmd > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: using > IOMMU type 1 (Type 1) > Oct 26 05:28:06 localhost.localdomain ovs-vswitchd[13799]: EAL: Ignore > mapping IO port bar(2) addr: ffc1 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: EAL: PCI device > 0000:8b:00.1 on NUMA socket 2 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: EAL: probe > driver: 8086:10fb rte_ixgbe_pmd > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00013|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.6.0 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00014|timeval|WARN|Unreasonably long 6567ms poll interval (232ms user, > 5958ms system) > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00015|timeval|WARN|faults: 9659 minor, 10 major > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00016|timeval|WARN|disk: 2384 reads, 0 writes > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00017|timeval|WARN|context switches: 19 voluntary, 62 involuntary > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]:
This message appears right after startup, which might explain it, since DPDK seems to take a long time to initialize (6 seconds in this case). Do you see any other messages like that after you try to operate? What is the problem that you see exactly? What is the configuration of the bridges? Can you send the output of ovs-vsctl show? Cascardo. > ovs|00018|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last > minute, last hour, hash=ec4c58ab: > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00019|coverage|INFO|bridge_reconfigure 0.0/sec 0.000/sec > 0.0000/sec total: 1 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00020|coverage|INFO|cmap_expand 0.0/sec 0.000/sec > 0.0000/sec total: 9 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00021|coverage|INFO|miniflow_malloc 0.0/sec 0.000/sec > 0.0000/sec total: 11 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00022|coverage|INFO|hmap_pathological 0.0/sec 0.000/sec > 0.0000/sec total: 3 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00023|coverage|INFO|hmap_expand 0.0/sec 0.000/sec > 0.0000/sec total: 646 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00024|coverage|INFO|txn_unchanged 0.0/sec 0.000/sec > 0.0000/sec total: 3 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00025|coverage|INFO|poll_create_node 0.0/sec 0.000/sec > 0.0000/sec total: 42 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00026|coverage|INFO|seq_change 0.0/sec 0.000/sec > 0.0000/sec total: 49 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00027|coverage|INFO|pstream_open 0.0/sec 0.000/sec > 0.0000/sec total: 1 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00028|coverage|INFO|stream_open 0.0/sec 0.000/sec > 0.0000/sec total: 1 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00029|coverage|INFO|util_xalloc 0.0/sec 0.000/sec > 0.0000/sec total: 11218 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00030|coverage|INFO|netdev_get_hwaddr 0.0/sec 0.000/sec > 0.0000/sec total: 2 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00031|coverage|INFO|netlink_received 0.0/sec 0.000/sec > 0.0000/sec total: 3 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00032|coverage|INFO|netlink_sent 0.0/sec 0.000/sec > 0.0000/sec total: 1 > Oct 26 05:28:07 localhost.localdomain ovs-vswitchd[13799]: > ovs|00033|coverage|INFO|89 events never hit > > Here are logs of DPDK: > > EAL: Detected 80 lcore(s) > EAL: No free hugepages reported in hugepages-1048576kB > EAL: Probing VFIO support... > EAL: VFIO support initialized > PMD: bnxt_rte_pmd_init() called for (null) > EAL: PCI device 0000:01:00.0 on NUMA socket 0 > EAL: probe driver: 8086:1521 rte_igb_pmd > EAL: PCI device 0000:01:00.1 on NUMA socket 0 > EAL: probe driver: 8086:1521 rte_igb_pmd > EAL: PCI device 0000:01:00.2 on NUMA socket 0 > EAL: probe driver: 8086:1521 rte_igb_pmd > EAL: PCI device 0000:01:00.3 on NUMA socket 0 > EAL: probe driver: 8086:1521 rte_igb_pmd > EAL: PCI device 0000:8b:00.0 on NUMA socket 2 > EAL: probe driver: 8086:10fb rte_ixgbe_pmd > EAL: using IOMMU type 1 (Type 1) > EAL: Ignore mapping IO port bar(2) addr: ffc1 > EAL: PCI device 0000:8b:00.1 on NUMA socket 2 > EAL: probe driver: 8086:10fb rte_ixgbe_pmd > Zone 0: name:<rte_eth_dev_data>, phys:0x98dced40, len:0x30100, > virt:0x7f5232fced40, socket_id:0, flags:0 > > -- > ----------------------------- > Sincerely, > Bo Sun > _______________________________________________ > discuss mailing list > discuss@openvswitch.org > http://openvswitch.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
