Thanks for locating the root cause.
I sent out a pair of patches that, together, I expect would at least
work around the problem in the common case:
https://patchwork.ozlabs.org/patch/722628/
https://patchwork.ozlabs.org/patch/722629/
Do you think you could try them out?
Thanks,
Ben.
On Wed, Feb 01, 2017 at 07:43:53AM +0000, Hexin Wang wrote:
> So this is a docker issue,
> http://stackissue.com/openshift/origin/sdn-set-veth-tx-queue-length-to-unblock-qos-11126.html
> Basically docker creates interface with an explicit 0 txqlen which causes the
> problem when QoS is switched to htb.
>
> Thanks.
>
> Hexin
>
>
>
>
> On 1/31/17, 6:18 PM, "ovs-discuss-boun...@openvswitch.org on behalf of Hexin
> Wang" <ovs-discuss-boun...@openvswitch.org on behalf of
> hexin.w...@nutanix.com> wrote:
>
> >Looks like the problem is because ovn is trying to touch eth0 qos. The
> >moment it changes qos_type from “linux-other” to “linux-htb”. Traffic stops
> >working.
> >
> >setup_qos (queue_map=0x7fffffffc230, egress_iface=0x63ae10 "eth0") at
> >ovn/controller/binding.c:240
> >240 int error = netdev_open(egress_iface, NULL, &netdev_phy);
> >(gdb) n
> >241 if (error) {
> >(gdb)
> >251 smap_init(&qdisc_details);
> >(gdb)
> >252 if (netdev_get_qos(netdev_phy, &qdisc_type, &qdisc_details) != 0
> >||
> >(gdb)
> >253 qdisc_type[0] == '\0') {
> >(gdb)
> >252 if (netdev_get_qos(netdev_phy, &qdisc_type, &qdisc_details) != 0
> >||
> >(gdb)
> >257 if (strcmp(qdisc_type, OVN_QOS_TYPE)) {
> >(gdb) print qdisc_type
> >$12 = 0x7ffff78ef8ec "linux-other"
> >(gdb) print qdisc_details
> >$13 = {map = {buckets = 0x7fffffffc258, one = 0x0, mask = 0, n = 0}}
> >(gdb) n
> >258 error = netdev_set_qos(netdev_phy, OVN_QOS_TYPE,
> >&qdisc_details);
> >(gdb) n
> >259 if (error) {
> >(gdb) print error
> >$14 = 0
> >
> >
> >Thanks.
> >
> >Hexin
> >
> >
> >
> >On 1/31/17, 6:12 PM, "Ritesh Rekhi" <ritesh.re...@nutanix.com> wrote:
> >
> >>+ Hexin
> >>
> >>-----Original Message-----
> >>From: Ben Pfaff [mailto:b...@ovn.org]
> >>Sent: Tuesday, January 31, 2017 3:49 PM
> >>To: Ritesh Rekhi <ritesh.re...@nutanix.com>
> >>Cc: ovs-discuss@openvswitch.org
> >>Subject: Re: [ovs-discuss] after setting ovs-vsctl set open .
> >>external-ids:ovn-encap-ip=<IP>, eth0 stops tansmitting any packets in
> >>docker container
> >>
> >>OK, what is OVN doing to the routing table or IP address configuration for
> >>the docker container?
> >>
> >>On Tue, Jan 31, 2017 at 11:12:40PM +0000, Ritesh Rekhi wrote:
> >>> That is what I am not able to find.If I have only 1 chassis connected
> >>> things work fine. Looks like OVN is doing something to the docker eth0
> >>> interface .Again this problem does't happen on a Vm.
> >>>
> >>> It only happens on a docker container (:.
> >>>
> >>> Thx
> >>> Ritesh
> >>>
> >>> -----Original Message-----
> >>> From: Ben Pfaff [mailto:b...@ovn.org]
> >>> Sent: Tuesday, January 31, 2017 1:38 PM
> >>> To: Ritesh Rekhi <ritesh.re...@nutanix.com>
> >>> Cc: ovs-discuss@openvswitch.org
> >>> Subject: Re: [ovs-discuss] after setting ovs-vsctl set open .
> >>> external-ids:ovn-encap-ip=<IP>, eth0 stops tansmitting any packets in
> >>> docker container
> >>>
> >>> OK, well then, what is happening to the networking stack that causes the
> >>> problem? OVN doesn't do anything to the host routing table or IP address
> >>> configuration, at least not intentionally. What's happening to the host?
> >>>
> >>> On Tue, Jan 31, 2017 at 06:36:10PM +0000, Ritesh Rekhi wrote:
> >>> > Hi Ben,
> >>> >
> >>> > Thx for the reply
> >>> >
> >>> > To repro this issue we don't need any entries in North bound and south
> >>> > bound database.
> >>> >
> >>> > Here are the steps
> >>> >
> >>> > 1) run my docker container
> >>> >
> >>> > docker run -itd --privileged -e CTL=10.4.163.249
> >>> > riteshrekhi/ovnstuff:2.6.90
> >>> >
> >>> > do docker attach and go inside docker container , there execute
> >>> > script test.sh which starts ovsdb-server, ovs-vswitchd and
> >>> > ovn-controller.After that it will connect to the south-db
> >>> >
> >>> > using command
> >>> >
> >>> > ovs-vsctl set open . external-ids:system-id=$uuid ovs-vsctl set open .
> >>> > external-ids:ovn-remote=tcp:$CTL:6642
> >>> > ovs-vsctl set open . external-ids:ovn-encap-type=vxlan ovs-vsctl
> >>> > set open . external-ids:ovn-encap-ip=$IP
> >>> >
> >>> >
> >>> > Right after you execute this script and if connection to south-db
> >>> > establishes, you won't be able to ping out of eth0
> >>> >
> >>> >
> >>> > 2) For version which is working you can do the same thing instead
> >>> > use riteshrekhi/ovnstuff
> >>> >
> >>> > Ritesh
> >>> >
> >>> > ===================================================================
> >>> >
> >>> > South-bound DB
> >>> >
> >>> > [root@controller ~]# ovn-sbctl show
> >>> > Chassis "0b50e2f7-7421-4536-9441-447af46c4cc4"
> >>> > hostname: "fe11f7697841"
> >>> > Encap vxlan
> >>> > ip: "10.4.150.6"
> >>> > options: {csum="true"}
> >>> > Chassis "0ebf5a6d-230a-4b9a-938e-976565d6993a"
> >>> > hostname: "23cc31791a67"
> >>> > Encap vxlan
> >>> > ip: "10.4.150.6"
> >>> > options: {csum="true"}
> >>> > Chassis "ahv-a-10.11.2.21-newname"
> >>> > hostname: "NTNX-16SM6C380006-A"
> >>> > Encap vxlan
> >>> > ip: "10.11.2.21"
> >>> > options: {csum="true"}
> >>> > Encap geneve
> >>> > ip: "10.11.2.21"
> >>> > options: {csum="true"}
> >>> > Chassis "2a083861-9841-4674-80ba-ed3c1bcf6918"
> >>> > hostname: "5af088595d12"
> >>> > Encap vxlan
> >>> > ip: "172.17.0.5"
> >>> > options: {csum="true"}
> >>> > Port_Binding "f4948a0f-6794-4140-9e4a-9616308467ee"
> >>> > Port_Binding "5c0e4397-7334-4f03-9ec3-fdc025588462"
> >>> > Port_Binding "f1fe3ff5-51ec-4790-ada8-c6e244e0cdd6"
> >>> > Port_Binding "f9e86d51-9b26-48f3-b2ad-8df492022a77"
> >>> > Chassis "de1cc8d2-fb4b-4819-822a-c68d921a84f3"
> >>> > hostname: "cfa5bc3de51c"
> >>> > Encap vxlan
> >>> > ip: "10.4.150.6"
> >>> > options: {csum="true"}
> >>> >
> >>> >
> >>> > North bound DB
> >>> >
> >>> > ovn-nbctl show
> >>> > switch 3a2d4174-27c9-46bc-9191-1d52ab16c057
> >>> > (neutron-f0a1215f-cd3c-4c0e-8653-95c7d115731a)
> >>> > port c1c22fbb-5928-41c8-aa1a-c5a746c22118
> >>> > addresses: ["fa:16:3e:6f:ed:ce 10.20.4.1"]
> >>> > port f1fe3ff5-51ec-4790-ada8-c6e244e0cdd6
> >>> > addresses: ["fa:16:3e:68:cb:45 10.20.3.3"]
> >>> > port 864f7948-f254-4df2-b1cb-39138293a80d
> >>> > addresses: ["fa:16:3e:a7:d1:38 10.20.3.1"]
> >>> > port 5c0e4397-7334-4f03-9ec3-fdc025588462
> >>> > addresses: ["fa:16:3e:4e:d2:0f 10.20.4.3"]
> >>> > switch 1462d793-7fc5-47a2-8ede-23860a61353e
> >>> > (neutron-3f99e567-26da-44e3-bc41-c90572f45028)
> >>> > port f4948a0f-6794-4140-9e4a-9616308467ee
> >>> > addresses: ["fa:16:3e:eb:85:2c 10.20.3.3"]
> >>> > port f9e86d51-9b26-48f3-b2ad-8df492022a77
> >>> > addresses: ["fa:16:3e:e8:28:75 10.20.4.3"]
> >>> > port 9504652a-6cd7-45a2-8edf-411e23fe4204
> >>> > addresses: ["fa:16:3e:de:8b:1d 10.20.4.1"]
> >>> > port f3abc8da-5a4a-4a83-977e-7bffc51003b9
> >>> > addresses: ["fa:16:3e:96:08:ea 10.20.3.1"]
> >>> > router bc57dc9c-1032-41fd-8768-f0c7536d9558
> >>> > (neutron-9bf118df-154b-4659-97f4-d0c2ce6394cd)
> >>> > port lrp-864f7948-f254-4df2-b1cb-39138293a80d
> >>> > mac: "fa:16:3e:a7:d1:38"
> >>> > networks: ["10.20.3.1/24"]
> >>> > port lrp-c1c22fbb-5928-41c8-aa1a-c5a746c22118
> >>> > mac: "fa:16:3e:6f:ed:ce"
> >>> > networks: ["10.20.4.1/24"]
> >>> > router 9cc399f6-60bc-4117-8f0a-941711697c8e
> >>> > (neutron-39c6e2b9-4e0b-4641-9ca0-2a4ccc06741c)
> >>> > router 7636ef03-96ad-470b-b343-49e75211a506
> >>> > (neutron-df27d70e-88d9-469f-ba83-e2e2034e389a)
> >>> > router 25cb2627-c45a-40af-a49d-f6fc727580b1
> >>> > (neutron-9b7bd489-d872-4ffc-b4e3-a142f99ab00f)
> >>> > port lrp-9504652a-6cd7-45a2-8edf-411e23fe4204
> >>> > mac: "fa:16:3e:de:8b:1d"
> >>> > networks: ["10.20.4.1/24"]
> >>> > port lrp-f3abc8da-5a4a-4a83-977e-7bffc51003b9
> >>> > mac: "fa:16:3e:96:08:ea"
> >>> > networks: ["10.20.3.1/24"]
> >>> >
> >>> > -----Original Message-----
> >>> > From: Ben Pfaff [mailto:b...@ovn.org]
> >>> > Sent: Tuesday, January 31, 2017 8:26 AM
> >>> > To: Ritesh Rekhi <ritesh.re...@nutanix.com>
> >>> > Cc: ovs-discuss@openvswitch.org
> >>> > Subject: Re: [ovs-discuss] after setting ovs-vsctl set open .
> >>> > external-ids:ovn-encap-ip=<IP>, eth0 stops tansmitting any packets
> >>> > in docker container
> >>> >
> >>> > On Mon, Jan 30, 2017 at 02:11:15AM +0000, Ritesh Rekhi wrote:
> >>> > > To test scalability of controller, I made a docker container which
> >>> > > simulates as chassis using the instructions given by Dustin
> >>> > > http://blog.spinhirne.com/2016/09/a-primer-on-ovn.html.
> >>> > >
> >>> > > Everything was working fine, I was able to connect to my controller
> >>> > > and can see the flows coming back to my controller. I was using
> >>> > > released version 2.6.0.
> >>> > >
> >>> > > For some patches wanted to get latest version and found that after I
> >>> > > upgrade to latest patch , my script doesn't work. Upon
> >>> > > troubleshooting found that when I give the last command which
> >>> > > establishes the connection to controller i.e ovs-vsctl set open .
> >>> > > external-ids:ovn-encap-ip=<IP> , eth0 stopped Transmitting any
> >>> > > packets on the container.
> >>> > >
> >>> > > I started poking around and found that commit
> >>> > > "1ea9b847135eddca39ed95c0811edd605f6ca114" introduced this issue. If
> >>> > > I use a commit before that things are working fine.
> >>> >
> >>> > OK... I'll need more information about your setup. To start, can you
> >>> > provide your OVN northbound and southbound databases?
> >_______________________________________________
> >discuss mailing list
> >disc...@openvswitch.org
> >https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
