On 12/3/20 2:01 PM, Odintsov Vladislav wrote:
> But neither IP nor system-id was changed. I've double-checked:
>
> ovn-controller 20.06.2:
>
> Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
> hostname: host.local
> Encap vxlan
> ip: "172.24.33.105"
> options: {csum="true"}
> Encap stt
> ip: "172.24.33.105"
> options: {csum="true"}
> Port_Binding eni-3E9901E0
> Port_Binding eni-35AFCD00
>
> # ovs-vsctl get open . external-ids:system-id
> "04540082-b5b5-4ab5-9901-03ed445c772d"
>
> # systemctl stop ovn-controller
>
> Chassis was deleted:
>
> # ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d
> ovn-sbctl: no row "04540082-b5b5-4ab5-9901-03ed445c772d" in table Chassis
>
> # yum update ovn-host -y
> # systemctl restart ovn-controller
>
> Chassis with same system-id and encap IPs was re-added:
>
> Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
> hostname: host.local
> Encap vxlan
> ip: "172.24.33.105"
> options: {csum="true"}
> Encap stt
> ip: "172.24.33.105"
> options: {csum="true"}
>
> But, there are no port_bindings, and in ovn-controller logs again transaction
> error:
>
> 2020-12-03T12:53:54.031Z|00035|binding|INFO|Claiming lport eni-3E9901E0 for
> this chassis.
> 2020-12-03T12:53:54.031Z|00036|binding|INFO|eni-3E9901E0: Claiming
> 0a:00:3e:99:01:e0 192.168.0.4
> 2020-12-03T12:53:54.031Z|00037|binding|INFO|Claiming lport eni-35AFCD00 for
> this chassis.
> 2020-12-03T12:53:54.031Z|00038|binding|INFO|eni-35AFCD00: Claiming
> 0a:00:35:af:cd:00 192.168.0.5
> 2020-12-03T12:53:54.041Z|00039|ovsdb_idl|WARN|transaction error:
> {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\"
> role \"ovn-controller\" prohibit modification of table
> \"Encap\".","error":"permission error"}
> 2020-12-03T12:53:54.042Z|00040|main|INFO|OVNSB commit failed, force recompute
> next time.
>
>
> Moreover, if I forcefully delete chassis, port claim successful, but after
> restart ovn-controller, promlem appears again:
>
> # ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d
>
> 2020-12-03T12:56:20.119Z|00045|main|INFO|OVNSB commit failed, force recompute
> next time.
> 2020-12-03T12:56:23.803Z|00046|binding|INFO|Claiming lport eni-3E9901E0 for
> this chassis.
> 2020-12-03T12:56:23.803Z|00047|binding|INFO|eni-3E9901E0: Claiming
> 0a:00:3e:99:01:e0 192.168.0.4
> 2020-12-03T12:56:23.803Z|00048|binding|INFO|Claiming lport eni-35AFCD00 for
> this chassis.
> 2020-12-03T12:56:23.803Z|00049|binding|INFO|eni-35AFCD00: Claiming
> 0a:00:35:af:cd:00 192.168.0.5
>
> # systemctl restart ovn-controller
>
> 2020-12-03T12:56:38.590Z|00001|vlog|INFO|opened log file
> /var/log/ovn/ovn-controller.log
> 2020-12-03T12:56:38.592Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock:
> connecting...
> 2020-12-03T12:56:38.592Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock:
> connected
> 2020-12-03T12:56:38.596Z|00004|main|INFO|OVS IDL reconnected, force recompute.
> 2020-12-03T12:56:38.600Z|00005|reconnect|INFO|ssl:x.x.x.x:6642: connecting...
> 2020-12-03T12:56:38.600Z|00006|main|INFO|OVNSB IDL reconnected, force
> recompute.
> 2020-12-03T12:56:38.645Z|00007|reconnect|INFO|ssl:x.x.x.x:6642: connected
> 2020-12-03T12:56:38.650Z|00008|ofctrl|INFO|unix:/run/openvswitch/br-int.mgmt:
> connecting to switch
> 2020-12-03T12:56:38.650Z|00009|rconn|INFO|unix:/run/openvswitch/br-int.mgmt:
> connecting...
> 2020-12-03T12:56:38.651Z|00010|rconn|INFO|unix:/run/openvswitch/br-int.mgmt:
> connected
> 2020-12-03T12:56:38.654Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt:
> connecting to switch
> 2020-12-03T12:56:38.654Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt:
> connecting...
> 2020-12-03T12:56:38.654Z|00011|binding|INFO|Claiming lport eni-35AFCD00 for
> this chassis.
> 2020-12-03T12:56:38.654Z|00012|binding|INFO|eni-35AFCD00: Claiming
> 0a:00:35:af:cd:00 192.168.0.5
> 2020-12-03T12:56:38.654Z|00013|binding|INFO|Claiming lport eni-3E9901E0 for
> this chassis.
> 2020-12-03T12:56:38.654Z|00014|binding|INFO|eni-3E9901E0: Claiming
> 0a:00:3e:99:01:e0 192.168.0.4
> 2020-12-03T12:56:38.655Z|00015|ovsdb_idl|WARN|transaction error:
> {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\"
> role \"ovn-controller\" prohibit modification of table
> \"Encap\".","error":"permission error"}
> 2020-12-03T12:56:38.655Z|00016|main|INFO|OVNSB commit failed, force recompute
> next time.
>
>
> Maybe, I just don’t understand your idea...
I see. I'm pretty sure it's related to this commit that tries to reuse
Encaps (and that's wrong because it doesn't work with RBAC):
https://github.com/ovn-org/ovn/commit/94a32fca2d2b825fece0ef5b1873459bd9857dd3
I'll try to fix it and update this thread.
Regards,
Dumitru
>
> Regards,
>
> Vladislav Odintsov
>
> On 03.12.2020, 15:38, "Dumitru Ceara" <[email protected]> wrote:
>
> Sorry, I removed the list by accident, readding ovs-dev.
>
> On 12/3/20 1:23 PM, Odintsov Vladislav wrote:
> > Hi Dumitru,
> >
> > This helped!
> >
> > Chassis destroyed, and port successfully claimed:
> >
> > # ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d
> >
> > 2020-12-03T12:20:41.222Z|6550427|main|INFO|OVNSB commit failed, force
> recompute next time.
> > 2020-12-03T12:20:42.922Z|6550428|binding|INFO|Claiming lport
> eni-3E9901E0 for this chassis.
> > 2020-12-03T12:20:42.922Z|6550429|binding|INFO|eni-3E9901E0: Claiming
> 0a:00:3e:99:01:e0 192.168.0.4
> > 2020-12-03T12:20:42.922Z|6550430|binding|INFO|Claiming lport
> eni-35AFCD00 for this chassis.
> > 2020-12-03T12:20:42.922Z|6550431|binding|INFO|eni-35AFCD00: Claiming
> 0a:00:35:af:cd:00 192.168.0.5
> >
> >
> > Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
> > hostname: host.local
> > Encap vxlan
> > ip: "X.X.X.X"
> > options: {csum="true"}
> > Encap stt
> > ip: "X.X.X.X"
> > options: {csum="true"}
> > Port_Binding eni-3E9901E0
> > Port_Binding eni-35AFCD00
> >
> >
> > But I don't understand what could go wrong? Problem appears right after
> fresh deploy.
>
> I guess the system-id of the chassis changed but the IP didn't. This is
> an issue that should be documented as the CMS should clear the stale
> chassis entries if RBAC is enabled [0].
>
> Regards,
> Dumitru
>
> [0]
> https://mail.openvswitch.org/pipermail/ovs-dev/2020-September/374653.html
>
> >
> >
> > Regards,
> >
> > Vladislav Odintsov
> > Lead System Engineer at Croc Cloud Development Team
> >
> > On 03.12.2020, 15:15, "Dumitru Ceara" <[email protected]> wrote:
> >
> > On 12/3/20 11:50 AM, Odintsov Vladislav wrote:
> > > Hi,
> > >
> > > It seems, that I see regression with claim port functionality in
> my OVN installation between v20.06.2 and latest branch-20.06 (78174ea) on
> cluster with enabled rbac.
> >
> > Hi Vladislav,
> >
> > >
> > > On v20.06.2 ovn-controller successfully claims port:
> > >
> > > # ovn-controller --version
> > > ovn-controller 20.06.2
> > > Open vSwitch Library 2.13.0
> > > OpenFlow versions 0x6:0x6
> > >
> > >
> > > 2020-12-02T18:25:28.787Z|00011|binding|INFO|Claiming lport
> eni-35AFCD00 for this chassis.
> > > 2020-12-02T18:25:28.787Z|00012|binding|INFO|eni-35AFCD00:
> Claiming 0a:00:35:af:cd:00 192.168.0.5
> > > 2020-12-02T18:25:28.787Z|00013|binding|INFO|Claiming lport
> eni-3E9901E0 for this chassis.
> > > 2020-12-02T18:25:28.787Z|00014|binding|INFO|eni-3E9901E0:
> Claiming 0a:00:3e:99:01:e0 192.168.0.4
> > >
> > >
> > > Transaction request:
> > >
> > > 2020-12-02T18:50:36.128Z|01605|jsonrpc|DBG|ssl:X.X.X.X:6642: send
> request, method="transact",
> params=["OVN_Southbound",{"where":[["_uuid","==",["uuid","4e9bd54c-f083-45cd-93d3-a65f4d20d688"]]],"row":{"chassis":["uuid","9d414bfc-da12-487e-80a0-5c1f2a98a05a"]},"op":"update","table":"Port_Binding"}],
> id=310
> > >
> > > # ovn-sbctl show | grep 04540082-b5b5-4ab5-9901-03ed445c772d -A 9
> > > Chassis "04540082-b5b5-4ab5-9901-03ed445c772d"
> > > hostname: host.local
> > > Encap vxlan
> > > ip: "Y.Y.Y.Y"
> > > options: {csum="true"}
> > > Encap stt
> > > ip: "Y.Y.Y.Y"
> > > options: {csum="true"}
> > > Port_Binding eni-3E9901E0
> > > Port_Binding eni-35AFCD00
> > >
> > >
> > > Then I run update OVN (doesn’t matter only ovn controller or full
> ovn installation):
> > >
> > > # ovn-controller --version
> > > ovn-controller 20.06.3
> > > Open vSwitch Library 2.13.0
> > > OpenFlow versions 0x6:0x6
> > >
> > > ovn-controller is unable to claim lport:
> > >
> > > 2020-12-02T18:53:35.309Z|00043|binding|INFO|Claiming lport
> eni-3E9901E0 for this chassis.
> > > 2020-12-02T18:53:35.309Z|00044|binding|INFO|eni-3E9901E0:
> Claiming 0a:00:3e:99:01:e0 192.168.0.4
> > > 2020-12-02T18:53:35.309Z|00045|binding|INFO|Claiming lport
> eni-DB28C420 for this chassis.
> > > 2020-12-02T18:53:35.309Z|00046|binding|INFO|eni-DB28C420:
> Claiming 0a:00:db:28:c4:20 192.168.0.6
> > > 2020-12-02T18:53:35.309Z|00047|binding|INFO|Claiming lport
> eni-35AFCD00 for this chassis.
> > > 2020-12-02T18:53:35.309Z|00048|binding|INFO|eni-35AFCD00:
> Claiming 0a:00:35:af:cd:00 192.168.0.5
> > > 2020-12-02T18:53:35.345Z|00049|ovsdb_idl|WARN|transaction error:
> {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\"
> role \"ovn-controller\" prohibit modification of table
> \"Encap\".","error":"permission error"}
> > > 2020-12-02T18:53:35.345Z|00050|main|INFO|OVNSB commit failed,
> force recompute next time.
> > >
> > >
> > > Transaction request (added encap table modification comparing to
> previous version):
> > >
> > > 2020-12-02T18:57:33.661Z|20500|jsonrpc|DBG|ssl:X.X.X.X:6642: send
> request, method="transact",
> params=["OVN_Southbound",{"where":[["_uuid","==",["uuid","9a397740-4072-4853-9b75-9cc120fe4b34"]]],"row":{"chassis":["uuid","e1436af9-4a15-4480-937e-7584e64033a3"]},"op":"update","table":"Port_Binding"},{"where":[["_uuid","==",["uuid","7ec10f55-c89a-4fd3-a2ab-8ac22f845c85"]]],"row":{"chassis_name":"04540082-b5b5-4ab5-9901-03ed445c772d"},"op":"update","table":"Encap"},{"where":[["_uuid","==",["uuid","256d47ca-ef69-4d75-b967-7ab19bd413a7"]]],"row":{"chassis_name":"04540082-b5b5-4ab5-9901-03ed445c772d"},"op":"update","table":"Encap"},{"where":[["_uuid","==",["uuid","34856b67-7f15-44d3-8071-e20ae0f6029f"]]],"row":{"chassis":["uuid","e1436af9-4a15-4480-937e-7584e64033a3"]},"op":"update","table":"Port_Binding"}],
> id=113
> > >
> > >
> > > I’ve configured rbac consulting with this instruction:
> https://docs.ovn.org/en/latest/tutorials/ovn-rbac.html
> > >
> > > Some rbac-related parameters:
> > >
> > > # ovn-sbctl list connection
> > > _uuid : 4940feb2-c4ae-47d9-ade7-6f25c26a2a71
> > > external_ids : {}
> > > inactivity_probe : []
> > > is_connected : false
> > > max_backoff : []
> > > other_config : {}
> > > read_only : false
> > > role : ""
> > > status : {}
> > > target : "pssl:16642"
> > >
> > > _uuid : ed9366ef-e352-4210-998f-655f648d638d
> > > external_ids : {}
> > > inactivity_probe : []
> > > is_connected : false
> > > max_backoff : []
> > > other_config : {}
> > > read_only : false
> > > role : ovn-controller
> > > status : {}
> > > target : "pssl:6642"
> > > # ovn-sbctl list rbac_role
> > > _uuid : 91e9fee1-4aff-4d94-93bf-d4c5119a0dd2
> > > name : ovn-controller
> > > permissions :
> {Chassis=4a0070bf-1327-4c4d-a7be-83cf91fa1e42,
> Encap=91da95b4-4eaf-4659-b803-789c72ea3fad,
> MAC_Binding=660466ef-f0f0-4e58-8be1-a6d16a640ef9,
> Port_Binding=046836f0-caf1-4d22-88b3-a1d9562d2b58,
> Service_Monitor=dabca251-6c8e-4953-8769-88f687285a60}
> > > # ovn-sbctl list rbac_permission
> > > _uuid : 91da95b4-4eaf-4659-b803-789c72ea3fad
> > > authorization : [chassis_name]
> > > insert_delete : true
> > > table : Encap
> > > update : [ip, options, type]
> > >
> > > _uuid : 046836f0-caf1-4d22-88b3-a1d9562d2b58
> > > authorization : [""]
> > > insert_delete : false
> > > table : Port_Binding
> > > update : [chassis]
> > >
> > > _uuid : dabca251-6c8e-4953-8769-88f687285a60
> > > authorization : [""]
> > > insert_delete : false
> > > table : Service_Monitor
> > > update : [status]
> > >
> > > _uuid : 660466ef-f0f0-4e58-8be1-a6d16a640ef9
> > > authorization : [""]
> > > insert_delete : true
> > > table : MAC_Binding
> > > update : [datapath, ip, logical_port, mac]
> > >
> > > _uuid : 4a0070bf-1327-4c4d-a7be-83cf91fa1e42
> > > authorization : [name]
> > > insert_delete : true
> > > table : Chassis
> > > update : [encaps, external_ids, nb_cfg,
> other_config, vtep_logical_switches]
> > >
> > >
> > > # ovs-vsctl get open . external-ids:system-id
> > > "04540082-b5b5-4ab5-9901-03ed445c772d"
> > > # ovs-vsctl get-ssl
> > > Private key:
> /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-privkey.pem
> > > Certificate:
> /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-cert.pem
> > > CA Certificate: /var/lib/openvswitch/pki/host/cacert.pem
> > > Bootstrap: false
> > > # openssl x509 -noout -subject -in
> /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-cert.pem
> > > subject= /C=US/ST=CA/O=Open vSwitch/OU=Open vSwitch
> certifier/CN=04540082-b5b5-4ab5-9901-03ed445c772d
> > >
> > > # ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d
> > > _uuid : e1436af9-4a15-4480-937e-7584e64033a3
> > > encaps : [256d47ca-ef69-4d75-b967-7ab19bd413a7,
> 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85]
> > > external_ids : {datapath-type="",
> iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan",
> is-interconn="false", ovn-bridge-mappings="", ovn-chassis-mac-mappings="",
> ovn-cms-options="", ovn-monitor-all="false"}
> > > hostname : host.local
> > > name : "04540082-b5b5-4ab5-9901-03ed445c772d"
> > > nb_cfg : 0
> > > other_config : {datapath-type="",
> iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan",
> is-interconn="false", ovn-bridge-mappings="", ovn-chassis-mac-mappings="",
> ovn-cms-options="", ovn-monitor-all="false"}
> > > transport_zones : []
> > > vtep_logical_switches: []
> > >
> > > # ovn-sbctl list encap 256d47ca-ef69-4d75-b967-7ab19bd413a7
> > > _uuid : 256d47ca-ef69-4d75-b967-7ab19bd413a7
> > > chassis_name : "04540082-b5b5-4ab5-9901-03ed445c772d"
> > > ip : "Y.Y.Y.Y"
> > > options : {csum="true"}
> > > type : stt
> > > # ovn-sbctl list encap 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85
> > > _uuid : 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85
> > > chassis_name : "04540082-b5b5-4ab5-9901-03ed445c772d"
> > > ip : "Y.Y.Y.Y"
> > > options : {csum="true"}
> > > type : vxlan
> > >
> > > Can anybody point me what could go wrong?
> > > Am I missing something?
> > >
> >
> > Could you please try to force ovn-controller to recreate the
> > Chassis/Chassis_private records after the update? Something like
> the
> > following for a chassis with
> name=04540082-b5b5-4ab5-9901-03ed445c772d:
> >
> > ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d
> > ovn-sbctl destroy chassis_private
> 04540082-b5b5-4ab5-9901-03ed445c772d
> >
> > Thanks,
> > Dumitru
> >
> > >
> > > Regards,
> > >
> > > Vladislav Odintsov
> > >
> > > _______________________________________________
> > > dev mailing list
> > > [email protected]
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > >
> >
> >
>
>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
