Hi Billy, Firstly, thank you for taking a look at the logs I sent yesterday.
Please see in-line. Please let me know if I miss anything. Thanks & Regards, PVMJ On Wed, Oct 24, 2018 at 3:36 PM O Mahony, Billy <[email protected]> wrote: > > > > > *From:* Manojawa Paritala [mailto:[email protected]] > *Sent:* Tuesday, October 23, 2018 5:37 PM > *To:* O Mahony, Billy <[email protected]> > *Cc:* [email protected]; [email protected]; Subba Rao > Kodavalla <[email protected]>; Song, Kee SangX <[email protected]>; > Srinivasa Goda <[email protected]>; Kris Rajana <[email protected]> > *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens > > > > Hi Billy, > > > > There are no br-flat1 entries in ovsdb. > > > > As suggested I increased the log level to debug and then tried the same > scenario again. Though the result was same (br-flat1 getting deleted), I > observed the below 2 issues (i assume). > > *[[BO'M]] You can just increase the log level for the bridge module (all > the extra revalidator debug is making it hard to read the logs.):* > > ovs-appctl vlog/set bridge:file:dbg > > > *Manoj :- Added the log level of debug to only bridge module and collected the logs. Attached zip file has the logs.* > *[[BO'M]] Also is there logging from the neutron agent. OvS should not be > removing the br-flat1 of it’s own accord. Something external is updating > ovsdb to remove it’s record. Is ovn in the loop here? Is there and > ovn-controller or similar process running on the host?* > > *I think this is important. As far as I know the decision to delete the > bridge will not be made by vswitchd. It will be something external that > will remove the record in ovsdb bridge table. If the neutron agent log does > not mention it is doing this then maybe check the ovsdb .log file (it may > not exist if it’s not configured on the ovsdb command line) – you’ll have > to check the ovsdb man pages and also set. The ovsdb-tool is showing just > the delete transaction so we need to find out where that delete request is > coming from.* > > > *Manoj :- I do not see any changes in the OVSDB logs after the configuration change. The attached zip file also contains neutron-ovs-agent logs.* > Issue-1 :- > > 1. Everything is up and running. That is all the bridges are displayed in > OVS and no issues in the logs. > > > > *[[BO'M]] can you add the output from vsctl show at this point so I can > see the desired post agent restart state.* > *Manoj :- Attached zip file contains the output of the ovs-vsctl show command before making changes to the neutron-ovs-agent and restarting the service.* > 2. I add the below entries in the OVS section of neutron's > openvswitch-agent.ini file and restart the respective service. > > > > datapath_type=netdev > > vhostuser_socket_dir=/var/run/openvswitch > > > > 3. As mentioned earlier, bridge br-flat1 is deleted. At this point of > time, observed the below. > > > > 3.1 br-int dtapath type changed from "system" to "netdev". > > 3.2. Not sure if it is an expected behavour, but there were MAC address > changed prints only for both br-flat1 & br-int. > > > > 2018-10-23T14:39:40.253Z|41205|in_band|DBG|br-int: remote MAC address > changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 > > 2018-10-23T14:39:41.347Z|41374|in_band|DBG|br-flat1: remote MAC address > changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 > > 2018-10-23T14:39:48.229Z|41852|in_band|DBG|br-int: remote MAC address > changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 > > 2018-10-23T14:39:55.032Z|42008|in_band|DBG|br-int: remote MAC address > changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 > > *[[BO'M]] The datapath type change is expected. The MAC address changes > I’m not sure.* > *Manoj :- Ok. Thank you for the clarification.* > > > 3.3 interface state of br-int & eth8 (attached to br-flat1) are down. > > *[[BO'M]] Can you copy the o/p from vsctl show again at this point.* > *Manoj :- Attached zip file contains the ovs-vsctl output after changing the nutron-ovs-agent parameters and restartong the service. Please note that this time we also observed that 2 bridges got deleted. br-flat1 & br-vlan. * > > > Attaching the debug logs of ovs-vswitch. > > > > > > Issue-2 :- > > 1. Everything is up and running. That is all the bridges are displayed in > OVS and no issues in the logs. I have one bridge named br0, which is of > netdev type and I have attached a dpdk port, vhost-user port. Everthing is > file. > > > > > > *[[BO'M]] When you say everything is fine br-flat1 is still missing right? > If that is the case lets stick with the missing bridge issue for now. > (There is actually quite a few issues in the vsctl output below that we > deal with after we figure out who deleted br-flat1).* > *Manoj :- Ok Sure. * > > > 2. Now, in the OVS section ofneutron's openvswitch-agent.ini file, to the > existing "bridge_mappings" key, I added an extra value "vlan1:br0". The new > key-value pair now is as below. I wanted to create a new network and then > map the bridge br0. So, I added this entry. > > > > bridge_mappings = flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0 > > > > 3. Now, when I restart the openvswitch-agent service, I observed that the > datapath type of br0 changed from netdev to system. In the ovs-vsctl show > output, I see the below. > > > > Bridge "br0" > > Controller "tcp:127.0.0.1:6633" > > is_connected: true > > fail_mode: secure > > Port "vhost-user-1" > > Interface "vhost-user-1" > > type: dpdkvhostuser > > error: "could not add network device vhost-user-1 to > ofproto (Invalid argument)" > > Port "br0" > > Interface "br0" > > type: internal > > Port "port0" > > Interface "port0" > > type: dpdk > > options: {dpdk-devargs="0000:af:00.1"} > > error: "could not add network device port0 to ofproto > (Invalid argument)" > > Port "phy-br0" > > Interface "phy-br0" > > type: patch > > options: {peer="int-br0"} > > > > Attaching the logs for this issue as well. > > > > > > Thanks & Regards, > > PVMJ > > > > On Tue, Oct 23, 2018 at 5:44 PM O Mahony, Billy <[email protected]> > wrote: > > Hi Manojawa, > > > > So is there any remaining entry br-flat entry in the ovsdb? Does it give > any clue to the reason – there may be a free-form ‘status’ or ‘info’ field > for that purpose. > > > > I can understand the situation where a bridge might get incorrectly > configured but I can’t understand why it is deleted by something other than > the agent. > > > > Maybe it tries to create the bridge, there is some error so it decides to > delete it. Are there more detailed log levels available for the agent? You > may be able to turn on more detailed logging for the bridge logging in OvS > too. > > > > /Billy. > > > > > > *From:* Manojawa Paritala [mailto:[email protected]] > *Sent:* Tuesday, October 23, 2018 12:16 PM > *To:* O Mahony, Billy <[email protected]> > *Cc:* [email protected]; [email protected] > *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens > > > > Hi Billy, > > > > Thank you for your reply. > > > > 1. Huge pages are properly set. Based on the dpdk configuration > dpdk-socket-mem="4096,4096", 8 pages were created under /dev/hugepages. > > 2. dpdk-p0 is not attached to br-flat1. Actually I defined the bridge as > br-flat1. > > 3. Yes, 'ovs-vsctl show' does not show br-flat1. As soon as I add the > below entries in openvswitch-agent.ini and restart the > neutron-openmvswitch-agent service, br-flat1 is getting deleted. I can see > that in the ovs-vswitch logs and also in the output of "ovsdb-tool -mmm > show-log" > > > > datapath_type=netdev > > vhostuser_socket_dir=/var/run/openvswitch > > > > 4. I do not see any errors in then neutron-openvswitch-agent logs, except > for the below which are displayed after the bridge is deleted. > > > > ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent > [req-99a234e3-c943-4234-8c4d-f0fdc594df8f - - - - -] Bridge br-flat1 for > physical network flat does not exist. Agent terminated! > > > > Thanks & Regards, > > PVMJ > > > > On Tue, Oct 23, 2018 at 3:06 PM O Mahony, Billy <[email protected]> > wrote: > > Hi, > > I don't see any errors relating to the dpdk interfaces. But it is also not > clear where the user-space drivers are bound and the hugepage memory is set > up. So double check those two items. > > Is the dpdk-p0 interface being attached to br-flat? Even if there are > issues with the dpdk port the bridge should not be deleted (at least not > automatically by OvS). > > Can you confirm with 'ovs-vsctl show' that the br-flat is actually not > present after the agent is restarted. And that the dpdk-p0 is not reporting > an error. > > What does the neutron-openmvswitch-agent logs say? > > Also run ovsdb-tool -mmm show-log which might give a clue as to when and > how br-flat is being modified. > > Regards, > Billy > > > -----Original Message----- > > From: [email protected] [mailto:ovs-dev- > > [email protected]] On Behalf Of Manojawa Paritala > > Sent: Monday, October 22, 2018 3:31 PM > > To: [email protected]; [email protected] > > Subject: [ovs-dev] Issues configuring OVS-DPDK in openstack queens > > > > Hello All, > > > > On a 3 node (one controller + 2 compute), we configured Openstack Queens > > using OSA with OVS. On all the nodes, we defined br-mgmt as linux > bridge, br- > > tun as private network and br-flat as external. > > Installation was successful and we could create networks and instances on > > Openstack. > > > > Below are the versions of the OVS packages used on each node. > > > > Controller :- openstack-vswitch - 2.9.0 > > Computes :- openstack-vswitch-dpdk - 2.9.0 (as we wanted to configure > dpdk on > > the compute hosts) > > > > The openstack-vswitch-dpdk 2.9.0 package that we installed had dpdk > version > > 17.11.3. When we tried to enable DPDK it failed with the below error. > > > > dpdk|ERR|DPDK not supported in this copy of Open vSwitch > > > > So, we downloaded the sources for dpdk 17.11.4 and openvswitch 2.9.2, > built > > openvswitch with dpdk as suggested in the below official link. > > No issues on Openstack or OVS. > > http://docs.openvswitch.org/en/latest/intro/install/dpdk/ > > > > Then, we added the below parameters to OVS and everything looked ok. > > No issues in Openstack or OVS. > > > > $ovs-vsctl get Open_vSwitch . other_config {dpdk-extra="-n 2", > dpdk-init="true", > > dpdk-lcore-mask="0x300000000000", dpdk-socket-mem="4096,4096", pmd- > > cpu-mask="0xf00003c0000", vhost-iommu-support="true"} > > > > Then on the compute node, in openvswitch_agent.ini file - OVS section, I > added > > the below (based on the link > > > https://docs.openstack.org/neutron/pike/contributor/internals/ovs_vhostuser.h > > tml > > ) > > and restarted neutron-openmvswitch-agent service. > > > > datapath_type=netdev > > vhostuser_socket_dir=/var/run/openvswitch > > > > After the above change, bridge br-flat is getting deleted from OVS. > > Attached are the logs after I restart the neutron-openmvswitch-agent > service on > > the compute now. Not sure what the issue is. > > > > Can any of you please let me know if we are missing anything? > > > > Best Regards, > > PVMJ > >
<<attachment: OVS-Issue-Logs-24th-OCT-2018.zip>>
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
