On Thu, Apr 23, 2020 at 9:59 PM Brendan Doyle <[email protected]> wrote:
>
> Hi,
>
> Sorry, this feels like another gotya FAQ type of question ..
>
> I have a logical router "lr_vcn1" and an "public" localnet switch
> "ls_external_vcn1"
> I configured a distributed router port on "lr_vcn1" and connect it to
> "ls_external_vcn1",
> schedule it and every thing works fine.
>
> But when I create a second distributed router port on "lr_vcn1" and
> connect it to the
> same switch and schedule it, it does not get scheduled?
>
> I feel like I'm missing something, perhaps I need another "public"
> switch or another
> localnet port on "ls_external_vcn1". But logically it feels like I
> should not need these
> as both distributed router ports are going out on the same physical
> network, I'd just
> like to spread the load across chassis. Some config
There is only one distributed gateway router port allowed per logical router.
If you see the ovn-northd log, you will notice a warning message
***
ovn_northd|WARN|Bad configuration: multiple ports with
redirect-chassis on same logical router lr_vcn1
***
Maybe you can achieve the same by adding another logical router and
connecting to the
same logical switches.
Thanks
Numan
>
>
> First distributed router port
> -------------------------------------
> ovn-nbctl list gateway_chassis
> _uuid : c28a305d-7376-4a3c-b72a-65fa2020a371
> chassis_name : ca-rain16
> external_ids : {}
> name : lr_vcn1_nat1-ls_external_vcn1-ca-rain16
> options : {}
> priority : 10
>
> _uuid : be0e8e6e-49be-444f-b1d1-6c43720f5d8b
> chassis_name : ca-rain06
> external_ids : {}
> name : lr_vcn1_nat1-ls_external_vcn1-ca-rain06
> options : {}
> priority : 15
>
> _uuid : a9ea6c87-7a37-41bb-9c21-2937794dc18d
> chassis_name : ca-rain05
> external_ids : {}
> name : lr_vcn1_nat1-ls_external_vcn1-ca-rain05
> options : {}
> priority : 20
>
> [rca-rain17 ]# ovn-sbctl show
> Chassis ca-rain06
> hostname: ca-rain06.us.oracle.com
> Encap geneve
> ip: "172.20.1.6"
> options: {csum="true"}
> Port_Binding "47433b54-ac10-42f1-ae84-cc6fbb580297"
> Port_Binding "06e85cca-867a-44fc-b2c1-be62f2fb06c0"
> Port_Binding "284195d2-9280-4334-900e-571ecd00327a"
> Chassis ca-rain16
> hostname: ca-rain16.us.oracle.com
> Encap geneve
> ip: "172.20.1.16"
> options: {csum="true"}
> Chassis ca-rain17
> hostname: ca-rain17.us.oracle.com
> Encap geneve
> ip: "172.20.1.17"
> options: {csum="true"}
> Chassis ca-rain05
> hostname: ca-rain05.us.oracle.com
> Encap geneve
> ip: "172.20.1.5"
> options: {csum="true"}
> Port_Binding "ce78fd2b-4c68-428c-baf1-71718e7f3871"
> Port_Binding "269089c4-9464-41ec-9f63-6b3804b34b07"
> Port_Binding "00bff7c0-2e2d-41ba-9485-3b5fa9801365"
> Port_Binding cr-lr_vcn1_nat1-ls_external_vcn1
> Port_Binding "1cb7d760-90b0-4201-9517-88cb2de31c79"
>
> Add the second distributed gateway port
> -------------------------------------------------------
> ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain05 21
> ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain06 16
> ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain16 11
>
> We can see it shows up as a gateway chassis:
>
> [ra-rain17 ~]# ovn-nbctl list gateway_chassis
> _uuid : d83fd9a5-f40c-4101-ad5a-9e7fe017232b
> chassis_name : ca-rain05
> external_ids : {}
> name : lr_vcn1_ngw2-ls_external_vcn1-ca-rain05
> options : {}
> priority : 21
>
> _uuid : 53919209-e7d7-414a-8774-3a0e5f7aacbb
> chassis_name : ca-rain06
> external_ids : {}
> name : lr_vcn1_ngw2-ls_external_vcn1-ca-rain06
> options : {}
> priority : 16
>
> _uuid : c28a305d-7376-4a3c-b72a-65fa2020a371
> chassis_name : ca-rain16
> external_ids : {}
> name : lr_vcn1_nat1-ls_external_vcn1-ca-rain16
> options : {}
> priority : 10
>
> _uuid : be0e8e6e-49be-444f-b1d1-6c43720f5d8b
> chassis_name : ca-rain06
> external_ids : {}
> name : lr_vcn1_nat1-ls_external_vcn1-ca-rain06
> options : {}
> priority : 15
>
> _uuid : a2444526-0fdc-4077-b3cb-1e5ba342e82c
> chassis_name : ca-rain16
> external_ids : {}
> name : lr_vcn1_ngw2-ls_external_vcn1-ca-rain16
> options : {}
> priority : 11
>
> _uuid : a9ea6c87-7a37-41bb-9c21-2937794dc18d
> chassis_name : ca-rain05
> external_ids : {}
> name : lr_vcn1_nat1-ls_external_vcn1-ca-rain05
> options : {}
> priority : 20
>
> But it has not been scheduled:
>
> ovn-sbctl show
> Chassis ca-rain06
> hostname: ca-rain06.us.oracle.com
> Encap geneve
> ip: "172.20.1.6"
> options: {csum="true"}
> Port_Binding "47433b54-ac10-42f1-ae84-cc6fbb580297"
> Port_Binding "06e85cca-867a-44fc-b2c1-be62f2fb06c0"
> Port_Binding "284195d2-9280-4334-900e-571ecd00327a"
> Chassis ca-rain16
> hostname: ca-rain16.us.oracle.com
> Encap geneve
> ip: "172.20.1.16"
> options: {csum="true"}
> Chassis ca-rain17
> hostname: ca-rain17.us.oracle.com
> Encap geneve
> ip: "172.20.1.17"
> options: {csum="true"}
> Chassis ca-rain05
> hostname: ca-rain05.us.oracle.com
> Encap geneve
> ip: "172.20.1.5"
> options: {csum="true"}
> Port_Binding "ce78fd2b-4c68-428c-baf1-71718e7f3871"
> Port_Binding "269089c4-9464-41ec-9f63-6b3804b34b07"
> Port_Binding "00bff7c0-2e2d-41ba-9485-3b5fa9801365"
> Port_Binding cr-lr_vcn1_nat1-ls_external_vcn1
> Port_Binding "1cb7d760-90b0-4201-9517-88cb2de31c79"
>
> The full nb config is:
> =================
> ovn-nbctl show
> switch 91e8b1b9-2f7f-4770-a454-ac8a1cf1dfb0 (ls_external_vcn1)
> port ls_external_vcn1-lr_vcn1_nat1
> type: router
> router-port: lr_vcn1_nat1-ls_external_vcn1
> port ls_external_vcn1-lr_vcn1_ngw2
> type: router
> router-port: lr_vcn1_ngw2-ls_external_vcn1
> port ln-ls_external_vcn1
> type: localnet
> addresses: ["unknown"]
> switch 10073c55-8f96-411f-a3b6-89b13389a084 (ls_vcn2)
> port 06e85cca-867a-44fc-b2c1-be62f2fb06c0
> addresses: ["52:54:00:2a:7b:49 192.17.1.6"]
> port ce78fd2b-4c68-428c-baf1-71718e7f3871
> addresses: ["52:54:00:d8:6e:eb 192.17.1.5"]
> port vcn2_subnet1-lr_vcn2
> type: router
> addresses: ["40:44:00:00:00:50"]
> router-port: lr_vcn2-vcn2_subnet1
> switch 0e58d3cd-c36a-4651-8b42-4821f653bcb2 (ls_vcn3)
> port vcn3_subnet1-lr_vcn3
> type: router
> addresses: ["40:44:00:00:00:60"]
> router-port: lr_vcn3-vcn3_subnet1
> port 269089c4-9464-41ec-9f63-6b3804b34b07
> addresses: ["52:54:00:30:38:35 192.16.1.5"]
> port 284195d2-9280-4334-900e-571ecd00327a
> addresses: ["52:54:00:02:55:96 192.16.1.6"]
> switch e89dea6b-0fbd-4a6e-aeef-8d6a213efb55 (ls_vcn1)
> port vcn1_subnet1-lr_vcn1
> type: router
> addresses: ["40:44:00:00:00:30"]
> router-port: lr_vcn1-vcn1_subnet1
> port vcn1_subnet2-lr_vcn1
> type: router
> addresses: ["40:44:00:00:00:40"]
> router-port: lr_vcn1-vcn1_subnet2
> port 00bff7c0-2e2d-41ba-9485-3b5fa9801365
> addresses: ["52:54:00:e6:4f:46 192.16.1.5"]
> port 1cb7d760-90b0-4201-9517-88cb2de31c79
> addresses: ["52:54:00:80:d0:c8 192.16.2.5"]
> port 47433b54-ac10-42f1-ae84-cc6fbb580297
> addresses: ["52:54:00:be:06:16 192.16.1.6"]
> router 6019ef0c-d5c3-428a-8fa2-4f7db3af2f4b (lr_vcn2)
> port lr_vcn2-vcn2_subnet1
> mac: "40:44:00:00:00:50"
> networks: ["192.17.1.1/24"]
> port lr_vcn2-lr_vcn1
> mac: "40:44:00:00:00:80"
> networks: ["100.64.30.2/10"]
> router 318562e0-6c1d-4bc7-9da0-d8ccbd63c0b6 (lr_vcn3)
> port lr_vcn3-vcn3_subnet1
> mac: "40:44:00:00:00:60"
> networks: ["192.16.1.1/24"]
> router 61d04449-adaf-406f-b280-ad5aa24f24ca (lr_vcn1)
> port lr_vcn1-lr_vcn2
> mac: "40:44:00:00:00:70"
> networks: ["100.64.30.1/10"]
> port lr_vcn1-vcn1_subnet2
> mac: "40:44:00:00:00:40"
> networks: ["192.16.2.1/24"]
> port lr_vcn1_nat1-ls_external_vcn1
> mac: "40:44:00:00:00:d0"
> networks: ["172.20.1.105/24"]
> gateway chassis: [ca-rain05 ca-rain06 ca-rain16]
> port lr_vcn1_ngw2-ls_external_vcn1
> mac: "40:44:00:00:00:f0"
> networks: ["172.20.1.107/24"]
> gateway chassis: [ca-rain05 ca-rain06 ca-rain16]
> port lr_vcn1-vcn1_subnet1
> mac: "40:44:00:00:00:30"
> networks: ["192.16.1.1/24"]
> nat 3abfbd91-eb71-4c14-b58f-ca28ccd0592c
> external ip: "172.20.1.205"
> logical ip: "192.16.1.0/24"
> type: "snat"
> nat b1b87b03-34b8-4cf9-b9e9-4d7ed3b0fe20
> external ip: "172.20.1.207"
> logical ip: "192.16.2.0/24"
> type: "snat"
>
> ovs-vsctl show on each hypervisor
> =============================
> [ca-rain05 ~]# ovs-vsctl show
> 169dc085-7224-42c3-b119-390b7d0fe450
> Bridge br-int
> fail_mode: secure
> Port vnet6
> Interface vnet6
> Port patch-br-int-to-ln-ls_external_vcn1
> Interface patch-br-int-to-ln-ls_external_vcn1
> type: patch
> options: {peer=patch-ln-ls_external_vcn1-to-br-int}
> Port ovn-ca-rai-0
> Interface ovn-ca-rai-0
> type: geneve
> options: {csum="true", key=flow, remote_ip="172.20.1.6"}
> bfd_status: {diagnostic="No Diagnostic",
> flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
> remote_state=up, state=up}
> Port br-int
> Interface br-int
> type: internal
> Port ovn-ca-rai-2
> Interface ovn-ca-rai-2
> type: geneve
> options: {csum="true", key=flow, remote_ip="172.20.1.17"}
> Port ovn-ca-rai-1
> Interface ovn-ca-rai-1
> type: geneve
> options: {csum="true", key=flow, remote_ip="172.20.1.16"}
> bfd_status: {diagnostic="No Diagnostic",
> flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
> remote_state=up, state=up}
> Port vnet4
> Interface vnet4
> Port vnet8
> Interface vnet8
> Port vnet2
> Interface vnet2
> Bridge br-ext
> Port patch-ln-ls_external_vcn1-to-br-int
> Interface patch-ln-ls_external_vcn1-to-br-int
> type: patch
> options: {peer=patch-br-int-to-ln-ls_external_vcn1}
> Port bond0
> Interface bond0
> Port br-ext
> Interface br-ext
> type: internal
> ovs_version: "2.13.90"
>
> [ca-rain06 ]# ovs-vsctl show
> dbcc7c2e-cf07-4052-b40d-d4f47f5560b0
> Bridge br-ext
> Port bond0
> Interface bond0
> Port br-ext
> Interface br-ext
> type: internal
> Port patch-ln-ls_external_vcn1-to-br-int
> Interface patch-ln-ls_external_vcn1-to-br-int
> type: patch
> options: {peer=patch-br-int-to-ln-ls_external_vcn1}
> Bridge br-int
> fail_mode: secure
> Port ovn-ca-rai-1
> Interface ovn-ca-rai-1
> type: geneve
> options: {csum="true", key=flow, remote_ip="172.20.1.17"}
> Port vnet3
> Interface vnet3
> Port vnet1
> Interface vnet1
> Port vnet5
> Interface vnet5
> Port br-int
> Interface br-int
> type: internal
> Port ovn-ca-rai-0
> Interface ovn-ca-rai-0
> type: geneve
> options: {csum="true", key=flow, remote_ip="172.20.1.16"}
> bfd_status: {diagnostic="No Diagnostic",
> flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
> remote_state=up, state=up}
> Port ovn-ca-rai-2
> Interface ovn-ca-rai-2
> type: geneve
> options: {csum="true", key=flow, remote_ip="172.20.1.5"}
> bfd_status: {diagnostic="No Diagnostic",
> flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
> remote_state=up, state=up}
> Port patch-br-int-to-ln-ls_external_vcn1
> Interface patch-br-int-to-ln-ls_external_vcn1
> type: patch
> options: {peer=patch-ln-ls_external_vcn1-to-br-int}
> ovs_version: "2.13.90"
>
> [ca-rain16]ovs-vsctl show
> d713dd73-1566-454c-9dd2-895b308955a7
> Bridge br-ext
> Port br-ext
> Interface br-ext
> type: internal
> Port bond0
> Interface bond0
> Port patch-ln-ls_external_vcn1-to-br-int
> Interface patch-ln-ls_external_vcn1-to-br-int
> type: patch
> options: {peer=patch-br-int-to-ln-ls_external_vcn1}
> Bridge br-int
> fail_mode: secure
> Port ovn-ca-rai-2
> Interface ovn-ca-rai-2
> type: geneve
> options: {csum="true", key=flow, remote_ip="172.20.1.5"}
> bfd_status: {diagnostic="No Diagnostic",
> flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
> remote_state=up, state=up}
> Port ovn-ca-rai-0
> Interface ovn-ca-rai-0
> type: geneve
> options: {csum="true", key=flow, remote_ip="172.20.1.6"}
> bfd_status: {diagnostic="No Diagnostic",
> flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic",
> remote_state=up, state=up}
> Port br-int
> Interface br-int
> type: internal
> Port ovn-ca-rai-1
> Interface ovn-ca-rai-1
> type: geneve
> options: {csum="true", key=flow, remote_ip="172.20.1.17"}
> Port patch-br-int-to-ln-ls_external_vcn1
> Interface patch-br-int-to-ln-ls_external_vcn1
> type: patch
> options: {peer=patch-ln-ls_external_vcn1-to-br-int}
> ovs_version: "2.13.90"
>
>
>
> The commands used to create each distributed router port
> ===================================================
>
> distributed router port 1:
> ------------------------------------
> ovn-nbctl ls-add ls_external_vcn1
> ovn-nbctl lsp-add ls_external_vcn1 ln-ls_external_vcn1
> ovn-nbctl lsp-set-type ln-ls_external_vcn1 localnet
> ovn-nbctl lsp-set-addresses ln-ls_external_vcn1 unknown
> ovn-nbctl lsp-set-options ln-ls_external_vcn1 network_name=physnet
>
> ovn-nbctl lrp-add lr_vcn1 lr_vcn1_nat1-ls_external_vcn1
> 40:44:00:00:00:d0 172.20.1.105/24
> ovn-nbctl lsp-add ls_external_vcn1 ls_external_vcn1-lr_vcn1_nat1
> ovn-nbctl lsp-set-type ls_external_vcn1-lr_vcn1_nat1 router
> ovn-nbctl lsp-set-addresses ls_external_vcn1-lr_vcn1_nat1 router
> ovn-nbctl lsp-set-options ls_external_vcn1-lr_vcn1_nat1
> router-port=lr_vcn1_nat1-ls_external_vcn1
>
> ovn-nbctl lr-nat-add lr_vcn1 snat 172.20.1.205 192.16.1.0/24
> ovn-nbctl lr-route-add lr_vcn1 "0.0.0.0/0" 172.20.1.1
> lr_vcn1_nat1-ls_external_vcn1
>
> ovn-nbctl lrp-set-gateway-chassis lr_vcn1_nat1-ls_external_vcn1 ca-rain05 20
> ovn-nbctl lrp-set-gateway-chassis lr_vcn1_nat1-ls_external_vcn1 ca-rain06 15
> ovn-nbctl lrp-set-gateway-chassis lr_vcn1_nat1-ls_external_vcn1 ca-rain16 10
>
> distributed router port 2:
> ------------------------------------
> ovn-nbctl lrp-add lr_vcn1 lr_vcn1_ngw2-ls_external_vcn1
> 40:44:00:00:00:f0 172.20.1.107/24
> ovn-nbctl lsp-add ls_external_vcn1 ls_external_vcn1-lr_vcn1_ngw2
> ovn-nbctl lsp-set-type ls_external_vcn1-lr_vcn1_ngw2 router
> ovn-nbctl lsp-set-addresses ls_external_vcn1-lr_vcn1_ngw2 router
> ovn-nbctl lsp-set-options ls_external_vcn1-lr_vcn1_ngw2
> router-port=lr_vcn1_ngw2-ls_external_vcn1
>
> ovn-nbctl lr-nat-add lr_vcn1 snat 172.20.1.207 192.16.2.0/24
>
> ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain05 21
> ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain06 16
> ovn-nbctl lrp-set-gateway-chassis lr_vcn1_ngw2-ls_external_vcn1 ca-rain16 11
>
> Any thoughts?
>
> Thanks
>
> Brendan
>
>
>
>
>
>
>
>
>
> _______________________________________________
> discuss mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
