Re: [ovs-discuss] Second distributed router port on a gateway not being scheduled to a chassis

[Brendan Doyle]

On 24/04/2020 14:19, Numan Siddique wrote:
On Thu, Apr 23, 2020 at 9:59 PM Brendan Doyle <brendan.do...@oracle.com> wrote:
Hi,

Sorry, this feels like another gotya FAQ type of question ..

I have a logical router "lr_vcn1" and an "public" localnet switch
"ls_external_vcn1"
I configured a distributed router port on "lr_vcn1" and connect it to
"ls_external_vcn1",
schedule it and every thing works fine.

But when I create a second distributed router port on "lr_vcn1" and
connect it to the
same switch and schedule it, it does not get scheduled?

I feel like I'm missing something, perhaps I need another "public"
switch or another
localnet port on "ls_external_vcn1". But logically it feels like I
should not need these
as both distributed router ports are going out on the same physical
network, I'd just
like to spread the load across chassis. Some config
There is only one distributed gateway router port allowed per logical router.
If you see the ovn-northd log, you will notice a warning message

Thanks, that's the conclusion I came to.
***
ovn_northd|WARN|Bad configuration: multiple ports with
redirect-chassis on same logical router lr_vcn1
***

Maybe you can achieve the same by adding another logical router and
connecting to the
same logical switches.

Yeah the problem is if both logical routers have a default route

ovn-nbctl lr-route-add lr0 "0.0.0.0/0" 172.20.1.1 lr0-public
ovn-nbctl lr-route-add lr1 "0.0.0.0/0" 172.20.1.1 lr1-public

And lr0 and lr1 are connected to the same ls_sw which has hosts in two
subnets connected to it's switch port. Then there are two routes to the 
world
and I can't control which one is taken.

Basically I need a logical switch per subnet and attach a specific 
distributed
logical gateway router to the specific subnets that I want to use that 
gateway.

Thanks



Thanks
Numan


First distributed router port
-------------------------------------
ovn-nbctl list gateway_chassis
_uuid               : c28a305d-7376-4a3c-b72a-65fa2020a371
chassis_name        : ca-rain16
external_ids        : {}
name                : lr_vcn1_nat1-ls_external_vcn1-ca-rain16
options             : {}
priority            : 10

_uuid               : be0e8e6e-49be-444f-b1d1-6c43720f5d8b
chassis_name        : ca-rain06
external_ids        : {}
name                : lr_vcn1_nat1-ls_external_vcn1-ca-rain06
options             : {}
priority            : 15

_uuid               : a9ea6c87-7a37-41bb-9c21-2937794dc18d
chassis_name        : ca-rain05
external_ids        : {}
name                : lr_vcn1_nat1-ls_external_vcn1-ca-rain05
options             : {}
priority            : 20

[rca-rain17 ]# ovn-sbctl show
Chassis ca-rain06
      hostname: ca-rain06.us.oracle.com
      Encap geneve
          ip: "172.20.1.6"
          options: {csum="true"}
      Port_Binding "47433b54-ac10-42f1-ae84-cc6fbb580297"
      Port_Binding "06e85cca-867a-44fc-b2c1-be62f2fb06c0"
      Port_Binding "284195d2-9280-4334-900e-571ecd00327a"
Chassis ca-rain16
      hostname: ca-rain16.us.oracle.com
      Encap geneve
          ip: "172.20.1.16"
          options: {csum="true"}
Chassis ca-rain17
      hostname: ca-rain17.us.oracle.com
      Encap geneve
          ip: "172.20.1.17"
          options: {csum="true"}
Chassis ca-rain05
      hostname: ca-rain05.us.oracle.com
      Encap geneve
          ip: "172.20.1.5"
          options: {csum="true"}
      Port_Binding "ce78fd2b-4c68-428c-baf1-71718e7f3871"
      Port_Binding "269089c4-9464-41ec-9f63-6b3804b34b07"
      Port_Binding "00bff7c0-2e2d-41ba-9485-3b5fa9801365"
      Port_Binding cr-lr_vcn1_nat1-ls_external_vcn1
      Port_Binding "1cb7d760-90b0-4201-9517-88cb2de31c79"

Add the second distributed gateway port
-------------------------------------------------------
ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain05 21
ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain06 16
ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain16 11

We can see it shows up as a gateway chassis:

[ra-rain17 ~]# ovn-nbctl list gateway_chassis
_uuid               : d83fd9a5-f40c-4101-ad5a-9e7fe017232b
chassis_name        : ca-rain05
external_ids        : {}
name                : lr_vcn1_ngw2-ls_external_vcn1-ca-rain05
options             : {}
priority            : 21

_uuid               : 53919209-e7d7-414a-8774-3a0e5f7aacbb
chassis_name        : ca-rain06
external_ids        : {}
name                : lr_vcn1_ngw2-ls_external_vcn1-ca-rain06
options             : {}
priority            : 16

_uuid               : c28a305d-7376-4a3c-b72a-65fa2020a371
chassis_name        : ca-rain16
external_ids        : {}
name                : lr_vcn1_nat1-ls_external_vcn1-ca-rain16
options             : {}
priority            : 10

_uuid               : be0e8e6e-49be-444f-b1d1-6c43720f5d8b
chassis_name        : ca-rain06
external_ids        : {}
name                : lr_vcn1_nat1-ls_external_vcn1-ca-rain06
options             : {}
priority            : 15

_uuid               : a2444526-0fdc-4077-b3cb-1e5ba342e82c
chassis_name        : ca-rain16
external_ids        : {}
name                : lr_vcn1_ngw2-ls_external_vcn1-ca-rain16
options             : {}
priority            : 11

_uuid               : a9ea6c87-7a37-41bb-9c21-2937794dc18d
chassis_name        : ca-rain05
external_ids        : {}
name                : lr_vcn1_nat1-ls_external_vcn1-ca-rain05
options             : {}
priority            : 20

But it has not been scheduled:

ovn-sbctl show
Chassis ca-rain06
      hostname: ca-rain06.us.oracle.com
      Encap geneve
          ip: "172.20.1.6"
          options: {csum="true"}
      Port_Binding "47433b54-ac10-42f1-ae84-cc6fbb580297"
      Port_Binding "06e85cca-867a-44fc-b2c1-be62f2fb06c0"
      Port_Binding "284195d2-9280-4334-900e-571ecd00327a"
Chassis ca-rain16
      hostname: ca-rain16.us.oracle.com
      Encap geneve
          ip: "172.20.1.16"
          options: {csum="true"}
Chassis ca-rain17
      hostname: ca-rain17.us.oracle.com
      Encap geneve
          ip: "172.20.1.17"
          options: {csum="true"}
Chassis ca-rain05
      hostname: ca-rain05.us.oracle.com
      Encap geneve
          ip: "172.20.1.5"
          options: {csum="true"}
      Port_Binding "ce78fd2b-4c68-428c-baf1-71718e7f3871"
      Port_Binding "269089c4-9464-41ec-9f63-6b3804b34b07"
      Port_Binding "00bff7c0-2e2d-41ba-9485-3b5fa9801365"
      Port_Binding cr-lr_vcn1_nat1-ls_external_vcn1
      Port_Binding "1cb7d760-90b0-4201-9517-88cb2de31c79"

The full nb config is:
=================
ovn-nbctl show
switch 91e8b1b9-2f7f-4770-a454-ac8a1cf1dfb0 (ls_external_vcn1)
      port ls_external_vcn1-lr_vcn1_nat1
          type: router
          router-port: lr_vcn1_nat1-ls_external_vcn1
      port ls_external_vcn1-lr_vcn1_ngw2
          type: router
          router-port: lr_vcn1_ngw2-ls_external_vcn1
      port ln-ls_external_vcn1
          type: localnet
          addresses: ["unknown"]
switch 10073c55-8f96-411f-a3b6-89b13389a084 (ls_vcn2)
      port 06e85cca-867a-44fc-b2c1-be62f2fb06c0
          addresses: ["52:54:00:2a:7b:49 192.17.1.6"]
      port ce78fd2b-4c68-428c-baf1-71718e7f3871
          addresses: ["52:54:00:d8:6e:eb 192.17.1.5"]
      port vcn2_subnet1-lr_vcn2
          type: router
          addresses: ["40:44:00:00:00:50"]
          router-port: lr_vcn2-vcn2_subnet1
switch 0e58d3cd-c36a-4651-8b42-4821f653bcb2 (ls_vcn3)
      port vcn3_subnet1-lr_vcn3
          type: router
          addresses: ["40:44:00:00:00:60"]
          router-port: lr_vcn3-vcn3_subnet1
      port 269089c4-9464-41ec-9f63-6b3804b34b07
          addresses: ["52:54:00:30:38:35 192.16.1.5"]
      port 284195d2-9280-4334-900e-571ecd00327a
          addresses: ["52:54:00:02:55:96 192.16.1.6"]
switch e89dea6b-0fbd-4a6e-aeef-8d6a213efb55 (ls_vcn1)
      port vcn1_subnet1-lr_vcn1
          type: router
          addresses: ["40:44:00:00:00:30"]
          router-port: lr_vcn1-vcn1_subnet1
      port vcn1_subnet2-lr_vcn1
          type: router
          addresses: ["40:44:00:00:00:40"]
          router-port: lr_vcn1-vcn1_subnet2
      port 00bff7c0-2e2d-41ba-9485-3b5fa9801365
          addresses: ["52:54:00:e6:4f:46 192.16.1.5"]
      port 1cb7d760-90b0-4201-9517-88cb2de31c79
          addresses: ["52:54:00:80:d0:c8 192.16.2.5"]
      port 47433b54-ac10-42f1-ae84-cc6fbb580297
          addresses: ["52:54:00:be:06:16 192.16.1.6"]
router 6019ef0c-d5c3-428a-8fa2-4f7db3af2f4b (lr_vcn2)
      port lr_vcn2-vcn2_subnet1
          mac: "40:44:00:00:00:50"
          networks: ["192.17.1.1/24"]
      port lr_vcn2-lr_vcn1
          mac: "40:44:00:00:00:80"
          networks: ["100.64.30.2/10"]
router 318562e0-6c1d-4bc7-9da0-d8ccbd63c0b6 (lr_vcn3)
      port lr_vcn3-vcn3_subnet1
          mac: "40:44:00:00:00:60"
          networks: ["192.16.1.1/24"]
router 61d04449-adaf-406f-b280-ad5aa24f24ca (lr_vcn1)
      port lr_vcn1-lr_vcn2
          mac: "40:44:00:00:00:70"
          networks: ["100.64.30.1/10"]
      port lr_vcn1-vcn1_subnet2
          mac: "40:44:00:00:00:40"
          networks: ["192.16.2.1/24"]
      port lr_vcn1_nat1-ls_external_vcn1
          mac: "40:44:00:00:00:d0"
          networks: ["172.20.1.105/24"]
          gateway chassis: [ca-rain05 ca-rain06 ca-rain16]
      port lr_vcn1_ngw2-ls_external_vcn1
          mac: "40:44:00:00:00:f0"
          networks: ["172.20.1.107/24"]
          gateway chassis: [ca-rain05 ca-rain06 ca-rain16]
      port lr_vcn1-vcn1_subnet1
          mac: "40:44:00:00:00:30"
          networks: ["192.16.1.1/24"]
      nat 3abfbd91-eb71-4c14-b58f-ca28ccd0592c
          external ip: "172.20.1.205"
          logical ip: "192.16.1.0/24"
          type: "snat"
      nat b1b87b03-34b8-4cf9-b9e9-4d7ed3b0fe20
          external ip: "172.20.1.207"
          logical ip: "192.16.2.0/24"
          type: "snat"

ovs-vsctl show on each hypervisor
=============================
[ca-rain05 ~]#  ovs-vsctl show
169dc085-7224-42c3-b119-390b7d0fe450
      Bridge br-int
          fail_mode: secure
          Port vnet6
              Interface vnet6
          Port patch-br-int-to-ln-ls_external_vcn1
              Interface patch-br-int-to-ln-ls_external_vcn1
                  type: patch
                  options: {peer=patch-ln-ls_external_vcn1-to-br-int}
          Port ovn-ca-rai-0
              Interface ovn-ca-rai-0
                  type: geneve
                  options: {csum="true", key=flow, remote_ip="172.20.1.6"}
                  bfd_status: {diagnostic="No Diagnostic",
flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
remote_state=up, state=up}
          Port br-int
              Interface br-int
                  type: internal
          Port ovn-ca-rai-2
              Interface ovn-ca-rai-2
                  type: geneve
                  options: {csum="true", key=flow, remote_ip="172.20.1.17"}
          Port ovn-ca-rai-1
              Interface ovn-ca-rai-1
                  type: geneve
                  options: {csum="true", key=flow, remote_ip="172.20.1.16"}
                  bfd_status: {diagnostic="No Diagnostic",
flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
remote_state=up, state=up}
          Port vnet4
              Interface vnet4
          Port vnet8
              Interface vnet8
          Port vnet2
              Interface vnet2
      Bridge br-ext
          Port patch-ln-ls_external_vcn1-to-br-int
              Interface patch-ln-ls_external_vcn1-to-br-int
                  type: patch
                  options: {peer=patch-br-int-to-ln-ls_external_vcn1}
          Port bond0
              Interface bond0
          Port br-ext
              Interface br-ext
                  type: internal
      ovs_version: "2.13.90"

[ca-rain06 ]#  ovs-vsctl show
dbcc7c2e-cf07-4052-b40d-d4f47f5560b0
      Bridge br-ext
          Port bond0
              Interface bond0
          Port br-ext
              Interface br-ext
                  type: internal
          Port patch-ln-ls_external_vcn1-to-br-int
              Interface patch-ln-ls_external_vcn1-to-br-int
                  type: patch
                  options: {peer=patch-br-int-to-ln-ls_external_vcn1}
      Bridge br-int
          fail_mode: secure
          Port ovn-ca-rai-1
              Interface ovn-ca-rai-1
                  type: geneve
                  options: {csum="true", key=flow, remote_ip="172.20.1.17"}
          Port vnet3
              Interface vnet3
          Port vnet1
              Interface vnet1
          Port vnet5
              Interface vnet5
          Port br-int
              Interface br-int
                  type: internal
          Port ovn-ca-rai-0
              Interface ovn-ca-rai-0
                  type: geneve
                  options: {csum="true", key=flow, remote_ip="172.20.1.16"}
                  bfd_status: {diagnostic="No Diagnostic",
flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
remote_state=up, state=up}
          Port ovn-ca-rai-2
              Interface ovn-ca-rai-2
                  type: geneve
                  options: {csum="true", key=flow, remote_ip="172.20.1.5"}
                  bfd_status: {diagnostic="No Diagnostic",
flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
remote_state=up, state=up}
          Port patch-br-int-to-ln-ls_external_vcn1
              Interface patch-br-int-to-ln-ls_external_vcn1
                  type: patch
                  options: {peer=patch-ln-ls_external_vcn1-to-br-int}
      ovs_version: "2.13.90"

[ca-rain16]ovs-vsctl show
d713dd73-1566-454c-9dd2-895b308955a7
      Bridge br-ext
          Port br-ext
              Interface br-ext
                  type: internal
          Port bond0
              Interface bond0
          Port patch-ln-ls_external_vcn1-to-br-int
              Interface patch-ln-ls_external_vcn1-to-br-int
                  type: patch
                  options: {peer=patch-br-int-to-ln-ls_external_vcn1}
      Bridge br-int
          fail_mode: secure
          Port ovn-ca-rai-2
              Interface ovn-ca-rai-2
                  type: geneve
                  options: {csum="true", key=flow, remote_ip="172.20.1.5"}
                  bfd_status: {diagnostic="No Diagnostic",
flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
remote_state=up, state=up}
          Port ovn-ca-rai-0
              Interface ovn-ca-rai-0
                  type: geneve
                  options: {csum="true", key=flow, remote_ip="172.20.1.6"}
                  bfd_status: {diagnostic="No Diagnostic",
flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
remote_state=up, state=up}
          Port br-int
              Interface br-int
                  type: internal
          Port ovn-ca-rai-1
              Interface ovn-ca-rai-1
                  type: geneve
                  options: {csum="true", key=flow, remote_ip="172.20.1.17"}
          Port patch-br-int-to-ln-ls_external_vcn1
              Interface patch-br-int-to-ln-ls_external_vcn1
                  type: patch
                  options: {peer=patch-ln-ls_external_vcn1-to-br-int}
      ovs_version: "2.13.90"



The commands used to create each distributed router port
===================================================

distributed router port 1:
------------------------------------
ovn-nbctl ls-add ls_external_vcn1
ovn-nbctl lsp-add ls_external_vcn1 ln-ls_external_vcn1
ovn-nbctl lsp-set-type ln-ls_external_vcn1 localnet
ovn-nbctl lsp-set-addresses ln-ls_external_vcn1 unknown
ovn-nbctl lsp-set-options ln-ls_external_vcn1 network_name=physnet

ovn-nbctl lrp-add lr_vcn1 lr_vcn1_nat1-ls_external_vcn1
40:44:00:00:00:d0 172.20.1.105/24
ovn-nbctl lsp-add ls_external_vcn1 ls_external_vcn1-lr_vcn1_nat1
ovn-nbctl lsp-set-type ls_external_vcn1-lr_vcn1_nat1 router
ovn-nbctl lsp-set-addresses ls_external_vcn1-lr_vcn1_nat1 router
ovn-nbctl lsp-set-options ls_external_vcn1-lr_vcn1_nat1
router-port=lr_vcn1_nat1-ls_external_vcn1

ovn-nbctl lr-nat-add lr_vcn1 snat 172.20.1.205 192.16.1.0/24
ovn-nbctl lr-route-add lr_vcn1 "0.0.0.0/0" 172.20.1.1
lr_vcn1_nat1-ls_external_vcn1

ovn-nbctl lrp-set-gateway-chassis lr_vcn1_nat1-ls_external_vcn1 ca-rain05 20
ovn-nbctl lrp-set-gateway-chassis lr_vcn1_nat1-ls_external_vcn1 ca-rain06 15
ovn-nbctl lrp-set-gateway-chassis lr_vcn1_nat1-ls_external_vcn1 ca-rain16 10

distributed router port 2:
------------------------------------
ovn-nbctl lrp-add lr_vcn1 lr_vcn1_ngw2-ls_external_vcn1
40:44:00:00:00:f0 172.20.1.107/24
ovn-nbctl lsp-add ls_external_vcn1 ls_external_vcn1-lr_vcn1_ngw2
ovn-nbctl lsp-set-type ls_external_vcn1-lr_vcn1_ngw2 router
ovn-nbctl lsp-set-addresses ls_external_vcn1-lr_vcn1_ngw2 router
ovn-nbctl lsp-set-options ls_external_vcn1-lr_vcn1_ngw2
router-port=lr_vcn1_ngw2-ls_external_vcn1

ovn-nbctl lr-nat-add lr_vcn1 snat 172.20.1.207 192.16.2.0/24

ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain05 21
ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain06 16
ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain16 11

Any thoughts?

Thanks

Brendan









_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss