Hi Billy, The issue of br-flat1 getting deleted is fixed now with the key-value pair of "datapath_types = netdev,system" in OVS neutron-openvswitch-agent.ini in the compute node.
Earlier, we used the key-value pair as "datapath_type = netdev". Please note the new key name we used is "datapath_types" and in the the value we mentioned "netdev,system". So, now our updated OVS section in neutron-openvswitch-agent.ini is as below. [OVS] bridge_mappings = flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0,vlan2:br1 local_ip = 192.168.5.14 datapath_types = netdev,system One issue that we see now is, when we add "vlan1:br0,vlan2:br1" to bridge_mappings and restart the neutron-openvswitch-agent.ini and openvsiwtch-agent services, in the "ovs-vsctl show" output, the ports in the bridges br0 & br1 have the below error. This issue is getting resolved, if I change the datapath_type of those bridges to netdev. Bridge "br1" Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "phy-br1" Interface "phy-br1" type: patch options: {peer="int-br1"} Port "vhost-user-5" Interface "vhost-user-5" type: dpdkvhostuser error: "could not add network device vhost-user-5 to ofproto (Invalid argument)" Port "port1" Interface "port1" type: dpdk options: {dpdk-devargs="0000:b1:00.1"} error: "Error attaching device '0000:b1:00.1' to DPDK" Port "vhost-user-6" Interface "vhost-user-6" type: dpdkvhostuser error: "could not add network device vhost-user-6 to ofproto (Invalid argument)" Port "br1" Interface "br1" type: internal Bridge "br0" Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "vhost-user-1" Interface "vhost-user-1" type: dpdkvhostuser error: "could not add network device vhost-user-1 to ofproto (Invalid argument)" Port "br0" Interface "br0" type: internal Port "port0" Interface "port0" type: dpdk options: {dpdk-devargs="0000:af:00.1"} error: "could not add network device port0 to ofproto (Invalid argument)" Port "phy-br0" Interface "phy-br0" type: patch options: {peer="int-br0"} Port "vhost-user-3" Interface "vhost-user-3" type: dpdkvhostuser error: "could not add network device vhost-user-3 to ofproto (Invalid argument)" Port "vhost-user-2" Interface "vhost-user-2" type: dpdkvhostuser error: "could not add network device vhost-user-2 to ofproto (Invalid argument)" Port "vhost-user-4" Interface "vhost-user-4" type: dpdkvhostuser error: "could not add network device vhost-user-4 to ofproto (Invalid argument)" Thanks & Regards, PVMJ On Wed, Oct 24, 2018 at 7:47 PM Manojawa Paritala <manojaw...@biarca.com> wrote: > Hi Billy, > > The reason for br-vlan getting deleted in my previous attempt was due to a > small misconfiguration in the openstack-agent.ini. I corrected it and > re-done the test. Attaching the new logs with this email. Please ignore the > earlier logs. > > Apologies for the confusion. > > Thanks & Regards, > PVMJ > > On Wed, Oct 24, 2018 at 7:26 PM Manojawa Paritala <manojaw...@biarca.com> > wrote: > >> Hi Billy, >> >> Firstly, thank you for taking a look at the logs I sent yesterday. >> >> Please see in-line. Please let me know if I miss anything. >> >> Thanks & Regards, >> PVMJ >> >> On Wed, Oct 24, 2018 at 3:36 PM O Mahony, Billy <billy.o.mah...@intel.com> >> wrote: >> >>> >>> >>> >>> >>> *From:* Manojawa Paritala [mailto:manojaw...@biarca.com] >>> *Sent:* Tuesday, October 23, 2018 5:37 PM >>> *To:* O Mahony, Billy <billy.o.mah...@intel.com> >>> *Cc:* ovs-disc...@openvswitch.org; ovs-dev@openvswitch.org; Subba Rao >>> Kodavalla <s...@biarca.com>; Song, Kee SangX <kee.sangx.s...@intel.com>; >>> Srinivasa Goda <sri...@biarca.com>; Kris Rajana <kr...@biarca.com> >>> *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens >>> >>> >>> >>> Hi Billy, >>> >>> >>> >>> There are no br-flat1 entries in ovsdb. >>> >>> >>> >>> As suggested I increased the log level to debug and then tried the same >>> scenario again. Though the result was same (br-flat1 getting deleted), I >>> observed the below 2 issues (i assume). >>> >>> *[[BO'M]] You can just increase the log level for the bridge module (all >>> the extra revalidator debug is making it hard to read the logs.):* >>> >>> ovs-appctl vlog/set bridge:file:dbg >>> >>> >>> >> *Manoj :- Added the log level of debug to only bridge module and >> collected the logs. Attached zip file has the logs.* >> >> >>> *[[BO'M]] Also is there logging from the neutron agent. OvS should not >>> be removing the br-flat1 of it’s own accord. Something external is updating >>> ovsdb to remove it’s record. Is ovn in the loop here? Is there and >>> ovn-controller or similar process running on the host?* >>> >>> *I think this is important. As far as I know the decision to delete the >>> bridge will not be made by vswitchd. It will be something external that >>> will remove the record in ovsdb bridge table. If the neutron agent log does >>> not mention it is doing this then maybe check the ovsdb .log file (it may >>> not exist if it’s not configured on the ovsdb command line) – you’ll have >>> to check the ovsdb man pages and also set. The ovsdb-tool is showing just >>> the delete transaction so we need to find out where that delete request is >>> coming from.* >>> >>> >>> >> *Manoj :- I do not see any changes in the OVSDB logs after the >> configuration change. The attached zip file also contains neutron-ovs-agent >> logs.* >> >> >>> Issue-1 :- >>> >>> 1. Everything is up and running. That is all the bridges are displayed >>> in OVS and no issues in the logs. >>> >>> >>> >>> *[[BO'M]] can you add the output from vsctl show at this point so I can >>> see the desired post agent restart state.* >>> >> *Manoj :- Attached zip file contains the output of the ovs-vsctl show >> command before making changes to the neutron-ovs-agent and restarting the >> service.* >> >> >>> 2. I add the below entries in the OVS section of neutron's >>> openvswitch-agent.ini file and restart the respective service. >>> >>> >>> >>> datapath_type=netdev >>> >>> vhostuser_socket_dir=/var/run/openvswitch >>> >>> >>> >>> 3. As mentioned earlier, bridge br-flat1 is deleted. At this point of >>> time, observed the below. >>> >>> >>> >>> 3.1 br-int dtapath type changed from "system" to "netdev". >>> >>> 3.2. Not sure if it is an expected behavour, but there were MAC address >>> changed prints only for both br-flat1 & br-int. >>> >>> >>> >>> 2018-10-23T14:39:40.253Z|41205|in_band|DBG|br-int: remote MAC address >>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 >>> >>> 2018-10-23T14:39:41.347Z|41374|in_band|DBG|br-flat1: remote MAC address >>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 >>> >>> 2018-10-23T14:39:48.229Z|41852|in_band|DBG|br-int: remote MAC address >>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 >>> >>> 2018-10-23T14:39:55.032Z|42008|in_band|DBG|br-int: remote MAC address >>> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 >>> >>> *[[BO'M]] The datapath type change is expected. The MAC address changes >>> I’m not sure.* >>> >> *Manoj :- Ok. Thank you for the clarification.* >> >>> >>> >>> 3.3 interface state of br-int & eth8 (attached to br-flat1) are down. >>> >>> *[[BO'M]] Can you copy the o/p from vsctl show again at this point.* >>> >> *Manoj :- Attached zip file contains the ovs-vsctl output after changing >> the nutron-ovs-agent parameters and restartong the service. Please note >> that this time we also observed that 2 bridges got deleted. br-flat1 & >> br-vlan. * >> >>> >>> >>> Attaching the debug logs of ovs-vswitch. >>> >>> >>> >>> >>> >>> Issue-2 :- >>> >>> 1. Everything is up and running. That is all the bridges are displayed >>> in OVS and no issues in the logs. I have one bridge named br0, which is of >>> netdev type and I have attached a dpdk port, vhost-user port. Everthing is >>> file. >>> >>> >>> >>> >>> >>> *[[BO'M]] When you say everything is fine br-flat1 is still missing >>> right? If that is the case lets stick with the missing bridge issue for >>> now. (There is actually quite a few issues in the vsctl output below that >>> we deal with after we figure out who deleted br-flat1).* >>> >> *Manoj :- Ok Sure. * >> >>> >>> >>> 2. Now, in the OVS section ofneutron's openvswitch-agent.ini file, to >>> the existing "bridge_mappings" key, I added an extra value "vlan1:br0". The >>> new key-value pair now is as below. I wanted to create a new network and >>> then map the bridge br0. So, I added this entry. >>> >>> >>> >>> bridge_mappings = flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0 >>> >>> >>> >>> 3. Now, when I restart the openvswitch-agent service, I observed that >>> the datapath type of br0 changed from netdev to system. In the ovs-vsctl >>> show output, I see the below. >>> >>> >>> >>> Bridge "br0" >>> >>> Controller "tcp:127.0.0.1:6633" >>> >>> is_connected: true >>> >>> fail_mode: secure >>> >>> Port "vhost-user-1" >>> >>> Interface "vhost-user-1" >>> >>> type: dpdkvhostuser >>> >>> error: "could not add network device vhost-user-1 to >>> ofproto (Invalid argument)" >>> >>> Port "br0" >>> >>> Interface "br0" >>> >>> type: internal >>> >>> Port "port0" >>> >>> Interface "port0" >>> >>> type: dpdk >>> >>> options: {dpdk-devargs="0000:af:00.1"} >>> >>> error: "could not add network device port0 to ofproto >>> (Invalid argument)" >>> >>> Port "phy-br0" >>> >>> Interface "phy-br0" >>> >>> type: patch >>> >>> options: {peer="int-br0"} >>> >>> >>> >>> Attaching the logs for this issue as well. >>> >>> >>> >>> >>> >>> Thanks & Regards, >>> >>> PVMJ >>> >>> >>> >>> On Tue, Oct 23, 2018 at 5:44 PM O Mahony, Billy < >>> billy.o.mah...@intel.com> wrote: >>> >>> Hi Manojawa, >>> >>> >>> >>> So is there any remaining entry br-flat entry in the ovsdb? Does it give >>> any clue to the reason – there may be a free-form ‘status’ or ‘info’ field >>> for that purpose. >>> >>> >>> >>> I can understand the situation where a bridge might get incorrectly >>> configured but I can’t understand why it is deleted by something other than >>> the agent. >>> >>> >>> >>> Maybe it tries to create the bridge, there is some error so it decides >>> to delete it. Are there more detailed log levels available for the agent? >>> You may be able to turn on more detailed logging for the bridge logging in >>> OvS too. >>> >>> >>> >>> /Billy. >>> >>> >>> >>> >>> >>> *From:* Manojawa Paritala [mailto:manojaw...@biarca.com] >>> *Sent:* Tuesday, October 23, 2018 12:16 PM >>> *To:* O Mahony, Billy <billy.o.mah...@intel.com> >>> *Cc:* ovs-disc...@openvswitch.org; ovs-dev@openvswitch.org >>> *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens >>> >>> >>> >>> Hi Billy, >>> >>> >>> >>> Thank you for your reply. >>> >>> >>> >>> 1. Huge pages are properly set. Based on the dpdk configuration >>> dpdk-socket-mem="4096,4096", 8 pages were created under /dev/hugepages. >>> >>> 2. dpdk-p0 is not attached to br-flat1. Actually I defined the bridge as >>> br-flat1. >>> >>> 3. Yes, 'ovs-vsctl show' does not show br-flat1. As soon as I add the >>> below entries in openvswitch-agent.ini and restart the >>> neutron-openmvswitch-agent service, br-flat1 is getting deleted. I can see >>> that in the ovs-vswitch logs and also in the output of "ovsdb-tool -mmm >>> show-log" >>> >>> >>> >>> datapath_type=netdev >>> >>> vhostuser_socket_dir=/var/run/openvswitch >>> >>> >>> >>> 4. I do not see any errors in then neutron-openvswitch-agent logs, >>> except for the below which are displayed after the bridge is deleted. >>> >>> >>> >>> ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent >>> [req-99a234e3-c943-4234-8c4d-f0fdc594df8f - - - - -] Bridge br-flat1 for >>> physical network flat does not exist. Agent terminated! >>> >>> >>> >>> Thanks & Regards, >>> >>> PVMJ >>> >>> >>> >>> On Tue, Oct 23, 2018 at 3:06 PM O Mahony, Billy < >>> billy.o.mah...@intel.com> wrote: >>> >>> Hi, >>> >>> I don't see any errors relating to the dpdk interfaces. But it is also >>> not clear where the user-space drivers are bound and the hugepage memory is >>> set up. So double check those two items. >>> >>> Is the dpdk-p0 interface being attached to br-flat? Even if there are >>> issues with the dpdk port the bridge should not be deleted (at least not >>> automatically by OvS). >>> >>> Can you confirm with 'ovs-vsctl show' that the br-flat is actually not >>> present after the agent is restarted. And that the dpdk-p0 is not reporting >>> an error. >>> >>> What does the neutron-openmvswitch-agent logs say? >>> >>> Also run ovsdb-tool -mmm show-log which might give a clue as to when and >>> how br-flat is being modified. >>> >>> Regards, >>> Billy >>> >>> > -----Original Message----- >>> > From: ovs-dev-boun...@openvswitch.org [mailto:ovs-dev- >>> > boun...@openvswitch.org] On Behalf Of Manojawa Paritala >>> > Sent: Monday, October 22, 2018 3:31 PM >>> > To: ovs-disc...@openvswitch.org; ovs-dev@openvswitch.org >>> > Subject: [ovs-dev] Issues configuring OVS-DPDK in openstack queens >>> > >>> > Hello All, >>> > >>> > On a 3 node (one controller + 2 compute), we configured Openstack >>> Queens >>> > using OSA with OVS. On all the nodes, we defined br-mgmt as linux >>> bridge, br- >>> > tun as private network and br-flat as external. >>> > Installation was successful and we could create networks and instances >>> on >>> > Openstack. >>> > >>> > Below are the versions of the OVS packages used on each node. >>> > >>> > Controller :- openstack-vswitch - 2.9.0 >>> > Computes :- openstack-vswitch-dpdk - 2.9.0 (as we wanted to configure >>> dpdk on >>> > the compute hosts) >>> > >>> > The openstack-vswitch-dpdk 2.9.0 package that we installed had dpdk >>> version >>> > 17.11.3. When we tried to enable DPDK it failed with the below error. >>> > >>> > dpdk|ERR|DPDK not supported in this copy of Open vSwitch >>> > >>> > So, we downloaded the sources for dpdk 17.11.4 and openvswitch 2.9.2, >>> built >>> > openvswitch with dpdk as suggested in the below official link. >>> > No issues on Openstack or OVS. >>> > http://docs.openvswitch.org/en/latest/intro/install/dpdk/ >>> > >>> > Then, we added the below parameters to OVS and everything looked ok. >>> > No issues in Openstack or OVS. >>> > >>> > $ovs-vsctl get Open_vSwitch . other_config {dpdk-extra="-n 2", >>> dpdk-init="true", >>> > dpdk-lcore-mask="0x300000000000", dpdk-socket-mem="4096,4096", pmd- >>> > cpu-mask="0xf00003c0000", vhost-iommu-support="true"} >>> > >>> > Then on the compute node, in openvswitch_agent.ini file - OVS section, >>> I added >>> > the below (based on the link >>> > >>> https://docs.openstack.org/neutron/pike/contributor/internals/ovs_vhostuser.h >>> > tml >>> > ) >>> > and restarted neutron-openmvswitch-agent service. >>> > >>> > datapath_type=netdev >>> > vhostuser_socket_dir=/var/run/openvswitch >>> > >>> > After the above change, bridge br-flat is getting deleted from OVS. >>> > Attached are the logs after I restart the neutron-openmvswitch-agent >>> service on >>> > the compute now. Not sure what the issue is. >>> > >>> > Can any of you please let me know if we are missing anything? >>> > >>> > Best Regards, >>> > PVMJ >>> >>> _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev