Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

[Daniel Alvarez Sanchez]

Thanks a lot Han! Great and swift work!
I'm testing them now, will let you know ASAP.

On Thu, Mar 1, 2018 at 8:39 AM, Han Zhou  wrote:

>
>
> On Mon, Feb 26, 2018 at 12:05 PM, Ben Pfaff  wrote:
> >
> > On Fri, Feb 23, 2018 at 03:51:28PM -0800, Han Zhou wrote:
> > > On Fri, Feb 23, 2018 at 2:17 PM, Ben Pfaff  wrote:
> > > >
> > > > On Tue, Feb 20, 2018 at 08:56:42AM -0800, Han Zhou wrote:
> > > > > On Tue, Feb 20, 2018 at 8:15 AM, Ben Pfaff  wrote:
> > > > > >
> > > > > > On Mon, Feb 19, 2018 at 11:33:11AM +0100, Daniel Alvarez Sanchez
> > > wrote:
> > > > > > > @Han, I can try rebase the patch if you want but that was
> > > > > > > basically renaming the Address_Set table and from Ben's
> > > > > > > comment, it may be better to keep the name. Not sure,
> > > > > > > however, how we can proceed to address Lucas' points in
> > > > > > > this thread.
> > > > > >
> > > > > > I wouldn't rename the table.  It sounds like the priority should
> be to
> > > > > > add support for sets of port names.  I thought that there was
> already
> > > a
> > > > > > patch for that to be rebased, but maybe I misunderstood.
> > > > >
> > > > > I feel it is better to add a new table for port group explicitly,
> and
> > > the
> > > > > column type can be a set of weak reference to Logical_Switch_Port.
> > > > > The benefits are:
> > > > > - Better data integrity: deleting a lport automatically deletes
> from the
> > > > > port group
> > > > > - No confusion about the type of records in a single table
> > > > > - Existing Address_Set mechanism will continue to be supported
> without
> > > any
> > > > > change
> > > > > - Furthermore, the race condition issue brought up by Lucas can be
> > > solved
> > > > > by supporting port-group in IP address match condition in
> > > ovn-controller,
> > > > > so that all addresses in the lports are used just like how
> AddressSet is
> > > > > used today. And there is no need for Neutron networking-ovn to use
> > > > > AddressSet any more. Since addresses are deduced from lports, the
> > > ordering
> > > > > of deleting/adding doesn't matter any more.
> > > > >
> > > > > How does this sound?
> > > >
> > > > Will we want sets of Logical_Router_Ports later?
> > > At least I don't see any use case in Neutron for router ports since
> > > Security Group is only for VIF ports.
> > >
> > > There is another tricky point I see while working on implementation. In
> > > Neutron, SG can be applied to ports across different networks, but in
> OVN
> > > lports works only on its own datapath, so in ovn-controller we need to
> be
> > > able to pickup related ports from the port-group when translating
> lflows
> > > for each datapath. I hope this is not an issue. Otherwise, Neutron
> plugin
> > > will have to divide the group of ports into sub-groups according to the
> > > lswitch they belong to, which would be a pain.
> >
> > I think that we can make ovn-controller gracefully tolerate that.
> >
> > Let's try this implementation.  I'm not excited about having a new table
> > for this purpose, but it sounds like the advantages may be worthwhile.
>
> Here are the patches: https://patchwork.ozlabs.org/
> project/openvswitch/list/?series=31165
>
> Thanks,
> Han
>
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] tcp stream is broken with userspace conntrack after VM migration

[Darrell Ball]

Thanks for the report Neo.

From:  on behalf of "Yangxiaoliang (Neo)" 

Date: Saturday, February 24, 2018 at 2:09 AM
To: "ovs-discuss@openvswitch.org" 
Subject: [ovs-discuss] tcp stream is broken with userspace conntrack after VM 
migration

Hi,

Recently we have met a TCP STREAM broken problem with userspace conntrack 
feature.  After reading the ovs maillist, I found a similar problem with mine:  
https://mail.openvswitch.org/pipermail/ovs-discuss/2018-January/045958.html
Our test case is not absolutely same with his, but the result is almost the 
same (The TCP stream is stopped, and this TCP stream will close finally with 
timeout reason. But ping is ok. ). Since these is no any conclusion about this 
TCP conntrack issue, so I describe my problem here again to check that if 
anyone have any idea.

Our test case is described as below:
We are testing VM migration based on ovs(2.7.3)+dpdk(16.11).  There an iperf3 
TCP_STREAM test (speed >= 8Gbps) that running between two VMs.
 (VM_client – host1)   <- iperf3 TCP STREAM test ->  (VM_server – host2)

During the TCP_STREAM  test is running, we migrate the iperf3 client VM over 
and over again (migrating the iperf3 server VM doesn’t reproduce this problem) .

I capture the packets on client and server, then found that  the client doesn’t 
receive the TCP ACK packet from the server.  The client retransmit  TCP packets 
over and over again, and the server replies ACK packet but the ACK packet was 
dropped in server side.

We read the TCP conntrack code and add some debugging log. Then we found that 
below branchs in tcp_conn_update  will not be true.

--branch 1:
if (SEQ_GEQ(src->seqhi, end)
/* Last octet inside other's window space */
&& SEQ_GEQ(seq, src->seqlo - (dst->max_win << dws))
/* Retrans: not more than one window back */
&& (ackskew >= -MAXACKWINDOW)
/* Acking not more than one reassembled fragment backwards */
&& (ackskew <= (MAXACKWINDOW << sws))
/* Acking not more than one window forward */
&& ((tcp_flags & TCP_RST) == 0 || orig_seq == src->seqlo
|| (orig_seq == src->seqlo + 1) || (orig_seq + 1 == src->seqlo))) {
/* Require an exact/+1 sequence match on resets when possible */

--branch 2:
} else if ((dst->state < CT_DPIF_TCPS_SYN_SENT
|| dst->state >= CT_DPIF_TCPS_FIN_WAIT_2
|| src->state >= CT_DPIF_TCPS_FIN_WAIT_2)
   && SEQ_GEQ(src->seqhi + MAXACKWINDOW, end)
   /* Within a window forward of the originating packet */
   && SEQ_GEQ(seq, src->seqlo - MAXACKWINDOW)) {
   /* Within a window backward of the originating packet */

--Our debugging log is added as below:
} else {
// debugging log here
return CT_UPDATE_INVALID;
}

--Debugging log sample
2018-02-11T16:15:24.902940+08:00|err|ovs-vswitchd[16113]|tcp_conn_update[399]|00056|conntrack_tcp(pmd8)|:
  ==
2018-02-11T16:15:24.903168+08:00|err|ovs-vswitchd[16113]|tcp_conn_update[405]|00057|conntrack_tcp(pmd8)|:
 reply=0, (tcp header: tcp_src=59138, tcp_dst=5201, tcp_seq=557256579, 
tcp_ack=1919922981, tcp_ctl=0x8010, tcp_winsz=229, tcp_csum=21303, 
tcp_urg=0), (CON SRC: src->state=2, src->seqlo=557324635, src->seqhi=557258028, 
src->max_win=229, src->wscale=64), (CON DST: dst->state=0, dst->seqlo=0, 
dst->seqhi=1923674917, dst->max_win=1, dst->wscale=64)
2018-02-11T16:15:24.903357+08:00|err|ovs-vswitchd[16113]|tcp_conn_update[410]|00058|conntrack_tcp(pmd8)|:
 +3 Check branch 1: SEQ_GEQ(src->seqhi, end)=** 1 **, SEQ_GEQ(seq, src->seqlo - 
(dst->max_win << dws))=** 0 **, (ackskew <= (MAXACKWINDOW << sws))=** 1 **, 
((tcp_flags & TCP_RST) == 0 || orig_seq == src->seqlo|| (orig_seq == src->seqlo 
+ 1) || (orig_seq + 1 == src->seqlo))=** 1 **
2018-02-11T16:15:24.903543+08:00|err|ovs-vswitchd[16113]|tcp_conn_update[414]|00059|conntrack_tcp(pmd8)|:
 +3 1st branch detail: (src->seqhi=557258028, end=557258027), (seq=557256579, 
src->seqlo - (dst->max_win << dws)=557308251), (ackskew=-1919922981, 
-MAXACKWINDOW=-67035), (ackskew=-1919922981, (MAXACKWINDOW << sws)=1098301440), 
(tcp_flags=0x0010, orig_seq=557256579, src->seqlo=557324635), sws=14, 
dws=14, orig_seq=557256579
2018-02-11T16:15:24.903729+08:00|err|ovs-vswitchd[16113]|tcp_conn_update[419]|00060|conntrack_tcp(pmd8)|:
 +3 Check branch 2: (dst->state < CT_DPIF_TCPS_SYN_SENT || dst->state >= 
CT_DPIF_TCPS_FIN_WAIT_2 || src->state >= CT_DPIF_TCPS_FIN_WAIT_2)=** 1 **, 
SEQ_GEQ(src->seqhi + MAXACKWINDOW, end)=** 1 **, SEQ_GEQ(seq, src->seqlo - 
MAXACKWINDOW)=** 0 **
2018-02-11T16:15:24.903936+08:00|err|ovs-vswitchd[16113]|tcp_conn_update[423]|00061|conntrack_tcp(pmd8)|:
 +3 2nd 

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

[Han Zhou]

On Mon, Feb 26, 2018 at 12:05 PM, Ben Pfaff  wrote:
>
> On Fri, Feb 23, 2018 at 03:51:28PM -0800, Han Zhou wrote:
> > On Fri, Feb 23, 2018 at 2:17 PM, Ben Pfaff  wrote:
> > >
> > > On Tue, Feb 20, 2018 at 08:56:42AM -0800, Han Zhou wrote:
> > > > On Tue, Feb 20, 2018 at 8:15 AM, Ben Pfaff  wrote:
> > > > >
> > > > > On Mon, Feb 19, 2018 at 11:33:11AM +0100, Daniel Alvarez Sanchez
> > wrote:
> > > > > > @Han, I can try rebase the patch if you want but that was
> > > > > > basically renaming the Address_Set table and from Ben's
> > > > > > comment, it may be better to keep the name. Not sure,
> > > > > > however, how we can proceed to address Lucas' points in
> > > > > > this thread.
> > > > >
> > > > > I wouldn't rename the table.  It sounds like the priority should
be to
> > > > > add support for sets of port names.  I thought that there was
already
> > a
> > > > > patch for that to be rebased, but maybe I misunderstood.
> > > >
> > > > I feel it is better to add a new table for port group explicitly,
and
> > the
> > > > column type can be a set of weak reference to Logical_Switch_Port.
> > > > The benefits are:
> > > > - Better data integrity: deleting a lport automatically deletes
from the
> > > > port group
> > > > - No confusion about the type of records in a single table
> > > > - Existing Address_Set mechanism will continue to be supported
without
> > any
> > > > change
> > > > - Furthermore, the race condition issue brought up by Lucas can be
> > solved
> > > > by supporting port-group in IP address match condition in
> > ovn-controller,
> > > > so that all addresses in the lports are used just like how
AddressSet is
> > > > used today. And there is no need for Neutron networking-ovn to use
> > > > AddressSet any more. Since addresses are deduced from lports, the
> > ordering
> > > > of deleting/adding doesn't matter any more.
> > > >
> > > > How does this sound?
> > >
> > > Will we want sets of Logical_Router_Ports later?
> > At least I don't see any use case in Neutron for router ports since
> > Security Group is only for VIF ports.
> >
> > There is another tricky point I see while working on implementation. In
> > Neutron, SG can be applied to ports across different networks, but in
OVN
> > lports works only on its own datapath, so in ovn-controller we need to
be
> > able to pickup related ports from the port-group when translating lflows
> > for each datapath. I hope this is not an issue. Otherwise, Neutron
plugin
> > will have to divide the group of ports into sub-groups according to the
> > lswitch they belong to, which would be a pain.
>
> I think that we can make ovn-controller gracefully tolerate that.
>
> Let's try this implementation.  I'm not excited about having a new table
> for this purpose, but it sounds like the advantages may be worthwhile.

Here are the patches:
https://patchwork.ozlabs.org/project/openvswitch/list/?series=31165

Thanks,
Han
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OpenVswitch

[Chris Boley]

I've been tinkering with OVS on Ubuntu 16.04 with the libvirt hypervisor.
I've gotten the XML based networks defined in the hypervisor and I've
gotten the host to understand it's interfacing with OVS.

So I'm hitting some sticking points that are starting to make me feel
well.. "thick".
I built the bridge.

"sudo ovs-vsctl add-br vbridge0"

I set up an external bond port on the bridge.
ovs-vsctl add-bond vbridge0 vbond0 eth0 eth1 lacp=active
other_config:lacp_time=fast trunks=2,3*#*#I'm kind of confused about
the trunks=2,3 part Do I really need that to pass the tagged frames to the
Cisco Switch?

That's brings up the bond "vbond0 tied to my vswitch0

My config in my cisco switch is a standard 2 port etherchannel.
with the good ole:
switchport trunk encapsulation dot1q
switchport mode trunk

Switch#sh mac ad
Switch#sh mac address-table | i Po1
   25254.0071.b1b6DYNAMIC Po1   << here's my VM
   10004.23d7.bd0cDYNAMIC Po1
   10004.23d7.bd0dDYNAMIC Po1



I have my libvirt network defined, when I do an "ovs-vsctl show" it looks
like this:
cboley@VMHOST:~$ sudo ovs-vsctl show
126a4b57-4837-42a9-95d6-d818b35e95bd
   Bridge "vbridge0"
   Port "vbond0"
   trunks: [2, 3]
   Interface "eth1"
   Interface "eth2"
   Port "vbridge0"
   Interface "vbridge0"
   type: internal
   Port "vnet0"
   tag: 2
   Interface "vnet0"
   ovs_version: "2.5.2"



What I'm "thick" about is the fact that unless I run the following commands
my bridge just won't work.
manually at the CLI of the VM host I must run.
ifconfig vbridge0 up
modprobe 8021q<<< I know why that's happening and it's an easy fix..
just stick it in the interfaces file.
vconfig add vbridge0 2
vconfig add vbridge0 3


After I do that my VM's have conectivity.

So after all that said, I know my vbridge0 doesn't come up without help..
I'm not an expert, I'm just muttling through info.
I've been using a lot of helpful info from blog.scottlowe.org BTW. His info
really saved my tail with libvirt.
Should I assume it's to just create the bridge and the bond in the
interfaces file manually like this instead?
OR can I create the vbridge0 under and OVS command and make it come up
automatically with OVS commands that I'm missing?

auto lo
iface lo inet loopback
post-up modprobe 8021q

auto eth0
iface eth0 inet dhcp


auto eth1
iface eth1 inet manual
  bond-master vbond0


auto eth2
iface eth2 inet manual
  bond-master vbond0


auto vbond0
allow-vnetbr0 vbond0
iface vbond0 inet manual
bond-mode 4
bond-miimon 100
bond-downdelay 200
bond-updelay 200
bond-lacp-rate 1
bond-slaves none
ovs_bridge vnetbr0
ovs_type OVSPort
pre-up ifconfig $IFACE up
post-down ifconfig $IFACE down
address 0.0.0.0


auto vnetbr0
allow-ovs vnetbr0
iface vnetbr0 inet manual
bridge_ports eth1 eth2
pre-up ifconfig $IFACE up
pre-up vconfig add vnetbr0 2
pre-up vconfig add vnetbr0 3
post-up ifconfig eth1 mtu 1520
post-up ifconfig eth1 promisc
post-up ethtool -G eth1 rx 4096
post-up ethtool -K eth1 rx off tx off sg off tso off ufo off gso off
gro off lro off rxvlan off txvlan off ntuple off rxhash off
post-up ethtool -N eth1 rx-flow-hash udp4 sdfn
post-up ethtool -N eth1 rx-flow-hash udp6 sdfn
post-up ethtool -C eth1 rx-usecs 1 rx-frames 0
post-up ethtool -C eth1 adaptive-rx off
post-up ifconfig eth2 mtu 1520
post-up ifconfig eth2 promisc
post-up ethtool -G eth2 rx 4096
post-up ethtool -K eth2 rx off tx off sg off tso off ufo off gso off
gro off lro off rxvlan off txvlan off ntuple off rxhash off
post-up ethtool -N eth2 rx-flow-hash udp4 sdfn
post-up ethtool -N eth2 rx-flow-hash udp6 sdfn
post-up ethtool -C eth2 rx-usecs 1 rx-frames 0
post-up ethtool -C eth2 adaptive-rx off
ovs_type OVSBridge
ovs_ports vnetbr0
post-down ifconfig $IFACE down

BTW.. I'm not reinvening anything and can't take credit for this idea. Some
really smart person (not me ;D ) posted this:
https://zcentric.com/2014/07/07/openvswitch-kvm-libvirt-ubuntu-vlans-the-right-way/


Any guidance on how I can or should proceed would be greatly appreciated!
Thanks in advance...






On Sun, Feb 25, 2018 at 3:31 PM, Chris Boley  wrote:

> Grant, truly awesome insights. Thank you. I'll be putting them to good
> use. Thanks a bunch for your time and consideration on this.
>
> As for the servers I'm working with: "SUPERMICRO SYS-5018A-MHN4 1U
> Rackmount Server"
> I'm seeing it on NewEgg right now for $729 You would of course need to add
> your own drives and RAM.
> Supermicro is now making a 5019 model. But I couldn't find that newer
> model that shipped with a case that had multiple drive bays.
> I supposed at that point if you're dealing with 12 cores you may want a
> separate storage device to hook up 

Re: [ovs-discuss] [OVN] Python IDL: handling modify operations in update2 notifications is slow

[Daniel Alvarez]

Thanks folks!

> On 28 Feb 2018, at 22:10, Ben Pfaff  wrote:
> 
>> On Tue, Feb 27, 2018 at 03:56:40PM +0100, Daniel Alvarez Sanchez wrote:
>> Hi all,
>> 
>> When working on the performance of the OVN OpenStack integration and
>> following up this thread [0].
>> 
>> I went a bit deeper trying to find out the root cause of the linear growth.
>> It happens that during the processing of the update2 messages, when there's
>> a 'modify' operation, the data is converted to JSON [1] and then we build a
>> Row from this JSON [2].
>> 
>> This seems pretty useless and, since the JSON document is not used anywhere
>> else, I just avoided that conversion. This conversion takes longer as the
>> number of elements in the row increases, which is the case of adding a new
>> address to the Address_Set table or adding new ACLs to the acl column in
>> Logical_Switch. This have a big impact at scale and it can be easily fixed.
>> 
>> After removing the extra conversions (I'll be submitting the patch in a
>> while), the time for creating ports from OpenStack remains almost constant
>> regardless of the amount of ports in the system as you can see in the
>> attached image and further in this blogpost [3] and it's cut down
>> considerably.
> 
> Great detective work!
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [OVN] Python IDL: handling modify operations in update2 notifications is slow

[Ben Pfaff]

On Tue, Feb 27, 2018 at 03:56:40PM +0100, Daniel Alvarez Sanchez wrote:
> Hi all,
> 
> When working on the performance of the OVN OpenStack integration and
> following up this thread [0].
> 
> I went a bit deeper trying to find out the root cause of the linear growth.
> It happens that during the processing of the update2 messages, when there's
> a 'modify' operation, the data is converted to JSON [1] and then we build a
> Row from this JSON [2].
> 
> This seems pretty useless and, since the JSON document is not used anywhere
> else, I just avoided that conversion. This conversion takes longer as the
> number of elements in the row increases, which is the case of adding a new
> address to the Address_Set table or adding new ACLs to the acl column in
> Logical_Switch. This have a big impact at scale and it can be easily fixed.
> 
> After removing the extra conversions (I'll be submitting the patch in a
> while), the time for creating ports from OpenStack remains almost constant
> regardless of the amount of ports in the system as you can see in the
> attached image and further in this blogpost [3] and it's cut down
> considerably.

Great detective work!
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] how to configure openvswitch connected to a host

[S hj]

Hello,

I want to configure the following topology so that host1 can ping host2.

host1openvswitch1-openvswitch2---host2



host1(10.0.1.10/24) - (10.0.1.20/24) openvswitch1(10.0.0.1/16) --
 --- (10.0.0.2/16) openvswitch2(10.0.2.20/24)  (10.0.2.10/24) host2

I want to configure openvswitches and bridges. But I do not know how to
configure openvswitches to forward packets toward hosts.


Do you have any suggestion?

Thank you in advance
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss