Interesting... it seems that the ovs-vsctl set bridge br-int datapath_type=netdev does take, but very soon after it is changed back to system. I'll try and figure out how that is happening.
Gabe > -----Original Message----- > From: Gabe Black > Sent: Wednesday, August 26, 2015 12:42 PM > To: 'Mooney, Sean K'; [email protected] > Subject: RE: duplicate option: of_interface > > Hi Sean, > > Yes, I did have the OVS_DATAPATH_TYPE set to netdev. Even after setting > the [ovs] datapath_type=netdev and trying to manually set the > datapath_type to netdev (ovs-vsctl set Bridge br-int > datapath_type=netdev), the br-int still shows system. > > The interesting thing is that the other bridges (br-ex and br-p6p1 are set to > netdev (automatically during stacking), but just br-int is system and > immutable it seems via manual ovs-vsctl commands). > > I'll keep playing around and see if there is some way to reconcile. > > Thanks, > Gabe > > > -----Original Message----- > > From: Mooney, Sean K [mailto:[email protected]] > > Sent: Wednesday, August 26, 2015 5:12 AM > > To: Gabe Black; [email protected] > > Subject: RE: duplicate option: of_interface > > > > I know the feeling. I think your horizon issue can be fixed by Running > > sudo iptables -F sudo iptables -X iptables based security groups do > > not work with vhost-user so this will have no effect on vm security > > > > I think the ovs-ofctl: br-int is not a bridge or a socket error is > > because the datapath_type on the bridge is not currently set to netdev. > > > > Looking at your previous error I noticed the follow command the > > datapath_type is being set to system. > > ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', > > > '--may-exist', 'add- br', 'br-int', '--', 'set', 'Bridge', 'br-int', > > > 'datapath_type=system'] > > > > > > I think the reason the bridge does not exists is because it has the > > wrong datapath type. > > You can validate this by running > > sudo ovs-vsctl list-br > > > > when using kernel ovs the output will be similar to this > > > > _uuid : 50c64fb3-3c0c-40f9-8e45-6384a99687aa > > controller : [] > > datapath_id : "0000b24fc650f940" > > datapath_type : system > > external_ids : {bridge-id=br-int} > > fail_mode : secure > > flood_vlans : [] > > flow_tables : {} > > ipfix : [] > > mirrors : [] > > name : br-int > > netflow : [] > > other_config : {} > > ports : [1e2aeb85-01a2-4f1d-a0ef-f4954acab3f2, b8c7a19e-c510- > 47c1- > > 874c-9d99ec6085f5, c370d053-6aea-4f16-a6bc-f54f1aabfcb5, > > e1a7ef91-53a4- 4222-9bf7-249888ad48ef, e917d939-44db-4661-9311- > bb68a6c132c4] > > protocols : ["OpenFlow10"] > > sflow : [] > > status : {} > > stp_enable : false > > > > when using ovs with dpdk the datapath_type should be netdev. > > > > If it is currently system and assuming you have > > OVS_DATAPATH_TYPE=netdev set in your local.conf this could be caused, > > as one of my patches to neutron merged Saturday and we have not > > updated our deployment code To add the appropriate configuration. > > > https://github.com/openstack/neutron/commit/5b708d5f0e9a5ddb4675148 > > 9a52665e673a5cb0b > > > > > > If you add > > [ovs] > > datapath_type = netdev > > to the ml2_conf.ini the q-agt will configure the dpdk netdev datapath > > instead of the system kernel datapath. > > > > For your running system you can run the following commands to set the > > datapath > > > > sudo ovs-vsctl set bridge <bridge name> datapath=netdev. > > > > This should be run for all ovs bridges. > > > > On huge table memory as the system is left running main memory gets > > fragment. > > when you start ovs with our service file it allocates hugempage memory > > in the kernel from the current free memory. > > > > When you compile dpdk there is a hugepage segment limit of 256 segment. > > If the dpdk application cannot allocated its hugepage memory in less > > segment then this limit the application will exit. > > You can adjust this limit by setting OVS_DPDK_MEM_SEGMENTS=<value> > in > > your local.conf The more often you stack and unstack between reboots > > the more fragmented your memory will be so for development I tend to > > set the OVS_DPDK_MEM_SEGMENTS quite high. > > > > Regards > > Sean. > > -----Original Message----- > > From: Gabe Black [mailto:[email protected]] > > Sent: Tuesday, August 25, 2015 9:45 PM > > To: Gabe Black; Mooney, Sean K; [email protected] > > Subject: RE: duplicate option: of_interface > > > > Sorry, that was premature. Apparently hugetable memory gets > > fragmented or leaks or something as the ovs-dpdk failed to start. > Rebooting fixed that. > > > > I still get many errors in the q-agt log file similar to this: > > ERROR neutron.agent.linux.utils [-] > > Command: ['ovs-ofctl', 'add-flows', 'br-int', '-'] Exit code: 1 > > Stdin: > > hard_timeout=0,idle_timeout=0,priority=0,table=0,cookie=0x0,actions=no > > r > > mal > > Stdout: > > Stderr: ovs-ofctl: br-int is not a bridge or a socket > > > > and > > > > ERROR neutron.agent.common.ovs_lib [req-ab1e1f65-d14c-4209-ac20- > > d29e450eda38 None None] Unable to execute ['ovs-ofctl', 'dump-flows', > > 'br- int', 'table=23']. Exception: > > Command: ['ovs-ofctl', 'dump-flows', 'br-int', 'table=23'] Exit code: > > 1 > > Stdin: > > Stdout: > > Stderr: ovs-ofctl: br-int is not a bridge or a socket > > > > But at least q-agt isn't dying. I remember why I hate fedora though.. > > Even though horizon says it is running and we turned the firewall to > > permissive, I can never access the website. Seems like something else > > is protecting or acting as a firewall. > > > > > > > > > -----Original Message----- > > > From: discuss [mailto:[email protected]] On Behalf Of > > > Gabe Black > > > Sent: Tuesday, August 25, 2015 2:06 PM > > > To: Mooney, Sean K; [email protected] > > > Subject: Re: [ovs-discuss] duplicate option: of_interface > > > > > > Thanks for the reply and hints on changes... Armed with that I > > > undid the changes from git commit 053bfc5a (neutron) and that > > > allowed the q-agt process to at least not die there with that > > > callstack. The commit message seemed to indicate it was for better > > > restarting/cleanup; so I hope it is relatively low impact to back out. > > > > > > However, it now dies because it is still unable to create the br-int > > > bridge: > > > > > > Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline', > > > '--format=json', '--', '--may-exist', 'add-br', 'br-int', '--', > > > 'set', 'Bridge', 'br-int', 'datapath_type=system']. > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl > > > Traceback (most recent call last): > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl File > > > "/opt/stack/neutron/neutron/agent/ovsdb/impl_vsctl.py", line 63, in > > > run_vsctl > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl > > > log_fail_as_error=False).rstrip() > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl File > > > "/opt/stack/neutron/neutron/agent/linux/utils.py", line 153, in execute > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl raise > > > RuntimeError(m) > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl > > > RuntimeError: > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl > Command: > > > ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', > > > '--may-exist', 'add- br', 'br-int', '--', 'set', 'Bridge', 'br-int', > > > 'datapath_type=system'] > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl Exit > > > code: -14 > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl Stdin: > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl Stdout: > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl Stderr: > > > 2015- 08-25T19:59:44Z|00002|fatal_signal|WARN|terminating with > > > signal > > > 14 (Alarm > > > clock) > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl > > > 2015-08-25 12:59:44.777 TRACE neutron.agent.ovsdb.impl_vsctl > > > > > > I noticed a difference between a multi-node setup and a single node, > > > where multi-node's controller has openvswitch and ovsdpdk for the > > > Q_ML2_PLUGIN_MECHANISM_DRIVERS. Is it perhaps related to that? > Or > > > should it be trying to create br-int as datapath_type netdev instead? > > > Just throwing things out there because I'm ignorant :-) I'll try > > > variations of that in hopes I get lucky. > > > > > > > > > Gabe > > > > > > > > > > > > > -----Original Message----- > > > > From: Mooney, Sean K [mailto:[email protected]] > > > > Sent: Tuesday, August 25, 2015 1:30 PM > > > > To: Gabe Black; [email protected] > > > > Subject: RE: duplicate option: of_interface > > > > > > > > Hi Gabe > > > > We have started to see that message in our ci since this weekend. > > > > We are currently investigating it but I belive a change has merged > > > > to neuton that we need to back port to Our agent. > > > > > > > > A lot of code has merged in the last 2 weeks as the code freeze > > > > for the liberty release is moday. > > > > The stable kilo branch should be unaffected but we are actively > > > > looking into this at present. > > > > > > > > Regards > > > > Sean. > > > > > > > > > > > > -----Original Message----- > > > > From: Gabe Black [mailto:[email protected]] > > > > Sent: Tuesday, August 25, 2015 7:50 PM > > > > To: [email protected] > > > > Cc: Mooney, Sean K > > > > Subject: duplicate option: of_interface > > > > > > > > I have followed the getting started guide > > > > (http://git.openstack.org/cgit/stackforge/networking-ovs- > > > > dpdk/tree/doc/source/getstarted.rst) on both fedora 21 and Ubuntu > > > > 15.04 to get a single-node set up with dpdk ovs. > > > > > > > > My local.conf file is identical to the one provided as the single > > > > node > > > template: > > > > http://git.openstack.org/cgit/stackforge/networking-ovs- > > > > dpdk/tree/doc/source/_downloads/local.conf.single_node > > > > > > > > I set HOST_IP_IFACE=eno1, HOST_IP=10.3.73.124, > > > > OVS_BRIDGE_MAPPINGS="default:br-enp4s0f0", and > > > > ML2_VLAN_RANGES=default:1000:1010 > > > > > > > > eno1 and associated IP is the interface/ip address of the server > > > > (i.e. what > > > we > > > > use to ssh to the box). enp4s0f0 is the 10G intel nic interface that > > > > will > > > > eventually be used for the data interface in a multi-node setup. > > > > Finally the vlan range was just arbitrarily chosen. > > > > > > > > Other than that, there isn't anything else modified other than > > > > following instructions of the getting started guide. However for > > > > both Fedora 21, and Ubuntu 15.04 (Ubuntu there were some mods that > > > > needed to take place like disabling apparmor, symlinking > > > > /var/run/openstack, and fixing ovs-dpdk-init > > > > script) result in the following error message in q-agt: > > > > > > > > Traceback (most recent call last): > > > > File "/usr/bin/networking-ovs-dpdk-agent", line 10, in <module> > > > > sys.exit(main()) > > > > File "/usr/lib/python2.7/site- > > > > > > packages/networking_ovs_dpdk/eventlet/ovs_dpdk_neutron_agent.py", > > > > line 20, in main > > > > agent_main.main() > > > > File "/usr/lib/python2.7/site- > > > > packages/networking_ovs_dpdk/agent/main.py", line 43, in main > > > > mod = importutils.import_module(mod_name) > > > > File > > > > "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", > > > > line 57, in import_module > > > > __import__(import_str) > > > > File "/usr/lib/python2.7/site- > > > > > > > > > > packages/networking_ovs_dpdk/agent/openflow/ovsdpdk_ofctl/main.py", > > > > line 17, in <module> > > > > from networking_ovs_dpdk.agent import ovs_dpdk_neutron_agent > > > > File "/usr/lib/python2.7/site- > > > > packages/networking_ovs_dpdk/agent/ovs_dpdk_neutron_agent.py", > > > line > > > > 47, in <module> > > > > from neutron.plugins.ml2.drivers.openvswitch.agent import > > > > ovs_dvr_neutron_agent > > > > File > > > > > > > > > > "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_ > > > > dvr_neutron_agent.py", line 29, in <module> > > > > cfg.CONF.import_group('AGENT', > > > > 'neutron.plugins.ml2.drivers.openvswitch.' > > > > File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line > > > > 2088, in import_group > > > > __import__(module_str) > > > > File > > > > > > > > > > "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/com > > > > mon/config.py", line 111, in <module> > > > > cfg.CONF.register_opts(ovs_opts, "OVS") > > > > File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line > > > > 1824, in __inner > > > > result = f(self, *args, **kwargs) > > > > File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line > > > > 1983, in register_opts > > > > self.register_opt(opt, group, clear_cache=False) > > > > File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line > > > > 1828, in __inner > > > > return f(self, *args, **kwargs) > > > > File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line > > > > 1967, in register_opt > > > > return group._register_opt(opt, cli) > > > > File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line > > > > 1345, in _register_opt > > > > if _is_opt_registered(self._opts, opt): > > > > File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line > > > > 574, in _is_opt_registered > > > > raise DuplicateOptError(opt.name) > > > > oslo_config.cfg.DuplicateOptError: duplicate option: of_interface > > > > q-agt failed to start > > > > > > > > I thought this error message was just because Ubuntu > > > > testing/support hasn't been fleshed out yet with ovs-dpdk, but > > > > then I got the exact same error on Fedora 21. I tried editing > > > > both > > > > > > > > > > /opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/comm > > > > on/config.py and /opt/stack/networking-ovs- > > > > dpdk/networking_ovs_dpdk/common/config.py to get past the error, > > but > > > > then there are complaints about not finding br-int... So I'm > > > > guessing that isn't the correct workaround. Anyone have any > > > > suggestions of what I might have misconfigured? > > > > > > > > Thank you for your help! > > > > Gabriel Black > > > > > > > > > > _______________________________________________ > > > discuss mailing list > > > [email protected] > > > http://openvswitch.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list [email protected] http://openvswitch.org/mailman/listinfo/discuss
