Hi Sean, Thanks for replying back, response inline.
On Mon, Nov 9, 2015 at 8:24 PM, Mooney, Sean K <[email protected]> wrote: > Hi > Can you provide some more information regarding your deployment? > > Can you check which kernel you are using. > > uname -a Linux ubuntu 3.16.0-50-generic #67~14.04.1-Ubuntu SMP Fri Oct 2 22:07:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux > > If you are using a 3.19 kernel changes to some locking code in the kennel broke synchronization dpdk2.0 and requires dpdk 2.1 to be used instead. > In general it is not advisable to use a 3.19 kernel with dpdk as it can lead to non-deterministic behavior. > > When devstack hangs can you connect with a second ssh session and run > sudo service ovs-dpdk status > and > ps aux | grep ovs > sudo service ovs-dpdk status sourcing config /opt/stack/logs/ovs-vswitchd.pid is not running Not all processes are running restart!!! 1 ubuntu@ubuntu:~/samta/devstack$ ps -ef | grep ovs root 13385 1 0 15:17 ? 00:00:00 /usr/sbin/ovsdb-server --detach --pidfile=/opt/stack/logs/ovsdb-server.pid --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options ubuntu 24451 12855 0 15:45 pts/0 00:00:00 grep --color=auto ovs > > When the deployment hangs at sudo ovs-vsctl br-set-external-id br-ex bridge-id br-ex > It usually means that the ovs-vswitchd process has exited. > The above result shows that ovs-vswitchd is not running. > This can happen for a number of reasons. > The vswitchd process may exit if it failed to allocate memory (due to memory fragmentation or lack of free hugepages) > if the ovs-vswitchd.log is not available can you check the the hugepage mount point was created in > /mnt/huge And that Iis mounted > Run > ls -al /mnt/huge > and > mount > ls -al /mnt/huge total 4 drwxr-xr-x 2 libvirt-qemu kvm 0 Nov 11 15:18 . drwxr-xr-x 3 root root 4096 May 15 00:09 .. ubuntu@ubuntu:~/samta/devstack$ mount /dev/mapper/ubuntu--vg-root on / type ext4 (rw,errors=remount-ro) proc on /proc type proc (rw,noexec,nosuid,nodev) sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) none on /sys/fs/cgroup type tmpfs (rw) none on /sys/fs/fuse/connections type fusectl (rw) none on /sys/kernel/debug type debugfs (rw) none on /sys/kernel/security type securityfs (rw) udev on /dev type devtmpfs (rw,mode=0755) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) none on /run/shm type tmpfs (rw,nosuid,nodev) none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755) none on /sys/fs/pstore type pstore (rw) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu) cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,relatime,net_cls) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event) cgroup on /sys/fs/cgroup/net_prio type cgroup (rw,relatime,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb) /dev/sda1 on /boot type ext2 (rw) systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd) hugetlbfs-kvm on /run/hugepages/kvm type hugetlbfs (rw,mode=775,gid=106) nodev on /mnt/huge type hugetlbfs (rw,uid=106,gid=106) nodev on /mnt/huge type hugetlbfs (rw,uid=106,gid=106) > then checkout how many hugepages are mounted > > cat /proc/meminfo | grep huge > cat /proc/meminfo | grep Huge AnonHugePages: 292864 kB HugePages_Total: 5 HugePages_Free: 5 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB > > the vswitchd process may also exit if it failed to initializes dpdk interfaces. > This can happen if no interface is compatible with the igb-uio or vfio-pci drivers > (note in the vfio-pci case all interface in the same iommu group must be bound to the vfio-pci driver and > The iommu must be enabled in the kernel command line with VT-d enabled in the bios) > > Can you check which interface are bound to the dpdk driver by running the following command > > /opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py --status > /opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py --status Network devices using DPDK-compatible driver ============================================ <none> Network devices using kernel driver =================================== 0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=p1p1 drv=ixgbe unused=igb_uio 0000:02:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=p4p1 drv=i40e unused=igb_uio 0000:03:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=p2p1 drv=i40e unused=igb_uio 0000:06:00.0 'I350 Gigabit Network Connection' if=em1 drv=igb unused=igb_uio *Active* 0000:06:00.1 'I350 Gigabit Network Connection' if=em2 drv=igb unused=igb_uio Other network devices ===================== 0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' unused=igb_uio > > Finally can you confim that ovs-dpdk compiled successfully by either check the xstack.log or > Checking for the BUILD_COMPLETE file in /opt/stack/ovs BUILD_COMPLETE exist in /opt/stack/ovs though its empty. > > Regards > sean > > > > > -----Original Message----- > From: Samta Rangare [mailto:[email protected]] > Sent: Monday, November 9, 2015 2:31 PM > To: Czesnowicz, Przemyslaw > Cc: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [networking-ovs-dpdk] > > Thanks for replying Przemyslaw, there is no ovs-vswitchd.log in /opt/stack/logs/. This is all contains inside (ovsdb-server.pid, screen). > > When I cancel stack .sh (ctr c), and try to rerun this $sudo ovs-vsctl br-set-external-id br-ex bridge-id br-ex it didnt hang, that means vSwitch was running isn't it ? > > But rerunning stack.sh after unstack hangs again. > > Thanks, > Samta > > On Mon, Nov 9, 2015 at 7:50 PM, Czesnowicz, Przemyslaw < [email protected]> wrote: >> Hi Samta, >> >> This usually means that the vSwitch is not running/has crashed. >> Can you check in /opt/stack/logs/ovs-vswitchd.log ? There should be an error msg there. >> >> Regards >> Przemek >> >>> -----Original Message----- >>> From: Samta Rangare [mailto:[email protected]] >>> Sent: Monday, November 9, 2015 1:51 PM >>> To: OpenStack Development Mailing List (not for usage questions) >>> Subject: [openstack-dev] [networking-ovs-dpdk] >>> >>> Hello Everyone, >>> >>> I am installing devstack with networking-ovs-dpdk. The local.conf >>> exactly looks like the one is available in /opt/stack/networking-ovs- >>> dpdk/doc/source/_downloads/local.conf.single_node. >>> So I believe all the necessary configuration will be taken care. >>> >>> However I am stuck at place where devstack is trying to set >>> external-id ($ sudo ovs-vsctl br-set-external-id br-ex bridge-id >>> br-ex). As soon as it hits at this place it's just hangs forever. I >>> tried commenting this line from >>> lib/neutron_plugin/ml2 (I know this is wrong) and then all services >>> came up except ovs-dpdk agent and ovs agent. >>> >>> BTW I am deploying it in ubuntu 14.04. Any pointer will be really helpful. >>> >>> Thanks, >>> Samta >>> >>> __________________________________________________________ >>> ________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: OpenStack-dev- >>> [email protected]?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
