[Yahoo-eng-team] [Bug 2056377] [NEW] Create Role button is not visible for default admin user
Public bug reported: When I navigate to the Identity->Roles page as the admin user, I see a list of the roles in the cluster, but no buttons to create/modify/delete the roles. I am using a charmed install of a Yoga cluster, and the default admin user (username: admin, domain: admin_domain, default keystone policies and default horizon policies, that is no overrides). I can create/modify/delete roles from the CLI. I thought this might have been a subtlety of the policy rules, so I tested including some Horizon policy overrides, which open up the policies to any user (setting every policy listed in the keystone/identity folders to "@"). This didn't change anything. To verify I was setting the policies correctly, I tested "identity:create_user": "!" and the "Create User" button disappeared in Horizon. This bug seems similar to [1] however I am using the default admin "power user" instead of adding domain roles. [1] https://bugs.launchpad.net/horizon/+bug/1775227 ** Affects: horizon Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/2056377 Title: Create Role button is not visible for default admin user Status in OpenStack Dashboard (Horizon): New Bug description: When I navigate to the Identity->Roles page as the admin user, I see a list of the roles in the cluster, but no buttons to create/modify/delete the roles. I am using a charmed install of a Yoga cluster, and the default admin user (username: admin, domain: admin_domain, default keystone policies and default horizon policies, that is no overrides). I can create/modify/delete roles from the CLI. I thought this might have been a subtlety of the policy rules, so I tested including some Horizon policy overrides, which open up the policies to any user (setting every policy listed in the keystone/identity folders to "@"). This didn't change anything. To verify I was setting the policies correctly, I tested "identity:create_user": "!" and the "Create User" button disappeared in Horizon. This bug seems similar to [1] however I am using the default admin "power user" instead of adding domain roles. [1] https://bugs.launchpad.net/horizon/+bug/1775227 To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/2056377/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1779978] Re: [fwaas] FWaaS instance stuck in PENDING_CREATE when devstack enable fwaas-v1
I am going to close this as fwaas-v1 has been deprecated. Please open a new bug if this also affects fwaas-v2. Thanks. ** Changed in: neutron Status: In Progress => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1779978 Title: [fwaas] FWaaS instance stuck in PENDING_CREATE when devstack enable fwaas-v1 Status in neutron: Invalid Bug description: When we deploy OpenStack by using devstack and enable FW v1 in local.conf "enable_service neutron-fwaas-v1", deploying process is successful, but when we create FW instance, instance will stuck in "PENDING_CREATE" status forever, I found a related bug https://bugs.launchpad.net/charm-neutron-gateway/+bug/1680164 , that only address for charm project, but problem still exist in devstack fwaas plugin, I add these options in my local environment and restart neutron services, then create FW instance, it will be in ACTIVE. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1779978/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2056199] Re: ``DvrDriver`` and ``OvnDriver`` incorrectly define distributed flag
Reviewed: https://review.opendev.org/c/openstack/neutron/+/911151 Committed: https://opendev.org/openstack/neutron/commit/06d48cb980d777514ffe9283eed0b59dcdb14345 Submitter: "Zuul (22348)" Branch:master commit 06d48cb980d777514ffe9283eed0b59dcdb14345 Author: Rodolfo Alonso Hernandez Date: Tue Mar 5 02:34:30 2024 + ``OvnDriver`` and ``DvrHaDriver`` to use "distributed_support" variable ``OvnDriver`` and ``DvrHaDriver`` classes were using an incorrect variable name to define the DVR support, that should be "distributed_support" instead of "dvr_support". Closes-Bug: #2056199 Change-Id: Id2ee080dde8cd094995e94564f2877a89e9cc5aa ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2056199 Title: ``DvrDriver`` and ``OvnDriver`` incorrectly define distributed flag Status in neutron: Fix Released Bug description: The class ``L3ServiceProvider`` defines the distributed support with the class variable "distributed_support" [1]. The classes ``DvrDriver`` and ``OvnDriver`` are using the variable "dvr_support" instead. The method to ensure a driver "ha" and "distributed" support uses "distributed_support" too [2]. [1]https://github.com/openstack/neutron/blob/c4c14f9589b54cc518564f1e1679898d2729c9e2/neutron/services/l3_router/service_providers/base.py#L57 [2]https://github.com/openstack/neutron/blob/c4c14f9589b54cc518564f1e1679898d2729c9e2/neutron/services/l3_router/service_providers/driver_controller.py#L273 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2056199/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2030976] Re: oslo notifications sending sensitive tokens
** Changed in: nova Status: Confirmed => Fix Released ** Changed in: ossa Status: Confirmed => Fix Released ** Changed in: oslo.messaging Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2030976 Title: oslo notifications sending sensitive tokens Status in Ironic: Fix Released Status in OpenStack Compute (nova): Fix Released Status in oslo.messaging: Fix Released Status in OpenStack Security Advisory: Fix Released Bug description: Hi, I have configured an OpenStack deployment to send Ironic service notifications using oslo_messaging_notifications[1] and noticed that Keystone tokens are being sent in the ['oslo.message']['_context_auth_token'] field of the message payload. - I have confirmed that auth token is leaked using both a Kafka and RabbitMQ backed - I have also confirmed that both messaging and messagingv2 options under oslo_messaging_notifications.driver are impacted[2] - I am using the Victoria version of Openstack and I have not confirmed if this has been patched on newer versions 1) https://docs.openstack.org/ironic/latest/admin/notifications.html 2) https://docs.openstack.org/ironic/victoria/configuration/sample-config.html To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/2030976/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2056366] [NEW] Neutron ml2/ovn does not exit when killed with SIGTERM
Public bug reported: When Neutron is killed with SIGTERM (like via systemctl), when using ML2/OVN neutron workers do not exit and instead are eventually killed with SIGKILL when the graceful timeout is reached (often around 1 minute). This is happening due to the signal handlers for SIGTERM. There are multiple issues. 1) oslo_service, ml2/ovn mech_driver, and ml2/ovo_rpc.py all call signal.signal(signal.SIGTERM, ...) overwriting each others signal handlers. 2) SIGTERM is handled in the main thread, and running blocking code there causes AssertionErrors in eventlet 3) The ml2/ovn cleanup code doesn't cause the process to end, so it interrupts the killing of the process oslo_service has a singleton SignalHandler class that solves all of these issues and we should use that instead of calling signal.signal() ourselves. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2056366 Title: Neutron ml2/ovn does not exit when killed with SIGTERM Status in neutron: New Bug description: When Neutron is killed with SIGTERM (like via systemctl), when using ML2/OVN neutron workers do not exit and instead are eventually killed with SIGKILL when the graceful timeout is reached (often around 1 minute). This is happening due to the signal handlers for SIGTERM. There are multiple issues. 1) oslo_service, ml2/ovn mech_driver, and ml2/ovo_rpc.py all call signal.signal(signal.SIGTERM, ...) overwriting each others signal handlers. 2) SIGTERM is handled in the main thread, and running blocking code there causes AssertionErrors in eventlet 3) The ml2/ovn cleanup code doesn't cause the process to end, so it interrupts the killing of the process oslo_service has a singleton SignalHandler class that solves all of these issues and we should use that instead of calling signal.signal() ourselves. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2056366/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2054324] Re: Iptables rule wrong if created a rule with protocol 4
Reviewed: https://review.opendev.org/c/openstack/neutron/+/911154 Committed: https://opendev.org/openstack/neutron/commit/cd1d191e335dca707ac9324fd81e642cb453a6cf Submitter: "Zuul (22348)" Branch:master commit cd1d191e335dca707ac9324fd81e642cb453a6cf Author: Brian Haley Date: Tue Mar 5 11:59:05 2024 -0500 Use the system-dependent string for IP protocol 4 iptables-save uses a system-dependent value, usually that found in /etc/protocols, when 'ipip' is given as the security group protocol. The intent is to always use the string value for IP protocol '4', as iptables-save has no '-n' flag to print values numerically. This updates a previous change (793dfb04d) that hard-coded that string to 'ipencap', which broke CentOS/Fedora, which uses 'ipv4'. For this reason we cannot hard-code anything in neutron-lib, this needs to be added dynamically, so this one-line change needs to stay here, and effectively closes the bug. Closes-bug: #2054324 Change-Id: Ic40b539c9ef5cfa4cbbd6575e19e653342e8342b ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2054324 Title: Iptables rule wrong if created a rule with protocol 4 Status in neutron: Fix Released Bug description: I followed this document to create security group rule for requirement "Allow ingress Protocol IPIP or 4" and I used "ipip" value https://docs.openstack.org/api-ref/network/v2/index.html#create-security-group-rule btw I expected my iptables rule on nova compute is "-A INPUT -s 172.16.2.0/24 -p 4 -j ACCEPT" but the runtime rule in kernel was "-A INPUT -s 172.16.2.0/24 -p ipip -j ACCEPT" and proto number was 94 not 4 // the rules output for one port Chain neutron-linuxbri-idf95737e-7 (1 references) pkts bytes target prot opt in out source destination 115K 219M RETURN all -- * * 0.0.0.0/00.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ 0 0 RETURN udp -- * * 0.0.0.0/0 172.16.2.165 udp spt:67 dpt:68 0 0 RETURN udp -- * * 0.0.0.0/0 255.255.255.255 udp spt:67 dpt:68 0 0 RETURN 94 -- * * 172.16.2.0/240.0.0.0/0 0 0 RETURN udp -- * * 172.16.2.0/240.0.0.0/0 udp dpt:8472 0 0 RETURN tcp -- * * 172.16.2.0/240.0.0.0/0 tcp dpt:4240 0 0 RETURN udp -- * * 172.16.2.0/240.0.0.0/0 udp multiport dports 3:32767 190 RETURN tcp -- * * 172.16.2.0/240.0.0.0/0 tcp dpt:10250 0 0 RETURN tcp -- * * 172.16.2.0/240.0.0.0/0 tcp dpt:4245 512 30720 RETURN tcp -- * * 172.16.2.0/240.0.0.0/0 tcp multiport dports 3:32767 0 0 RETURN icmp -- * * 172.16.2.0/240.0.0.0/0 0 0 RETURN udp -- * * 172.16.2.0/240.0.0.0/0 udp dpt:4789 0 0 RETURN tcp -- * * 172.16.2.0/240.0.0.0/0 tcp dpt:4244 2 142 RETURN tcp -- * * 172.16.2.0/240.0.0.0/0 tcp dpt:179 0 0 DROP all -- * * 0.0.0.0/00.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ 14 2122 neutron-linuxbri-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0/* Send unmatched traffic to the fallback chain. */ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2054324/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2008943] Fix included in openstack/neutron xena-eom
This issue was fixed in the openstack/neutron xena-eom release. ** Changed in: cloud-archive/xena Status: Triaged => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2008943 Title: OVN DB Sync utility cannot find NB DB Port Group Status in Ubuntu Cloud Archive: Fix Released Status in Ubuntu Cloud Archive ussuri series: Fix Committed Status in Ubuntu Cloud Archive victoria series: Fix Released Status in Ubuntu Cloud Archive wallaby series: Fix Released Status in Ubuntu Cloud Archive xena series: Fix Released Status in neutron: In Progress Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Focal: Fix Released Bug description: Runtime exception: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_aa9f203b_ec51_4893_9bda_cfadbff9f800 can occure while performing database sync between Neutron db and OVN NB db using neutron-ovn-db-sync-util. This exception occures when the `sync_networks_ports_and_dhcp_opts()` function ends up implicitly creating a new default security group for a tenant/project id. This is normally ok but the problem is that `sync_port_groups` was already called and thus the port_group does not exists in NB DB. When the `sync_acls()` is called later there is no port group found and exception occurs. Quick way to reproduce on ML2/OVN: - openstack project create test_project - openstack create network --project test_project test_network - openstack port delete $(openstack port list --network test_network -c ID -f value) # since this is an empty network only the metadata port should get listed and subsequently deleted - openstack security group delete test_project So now that you have a network without a metadata port in it and no default security group for the project/tenant that this network belongs to run neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn- neutron_sync_mode migrate The exeption should occur Here is a more realistic scenario how we can run into this with ML2/OVS to ML2/OVN migration. I am also including why the code runs into it. 1. ML2/OVS enviroment with a network but no default security group for the project/tenant associated with the network 2. Perform ML2/OVS to ML2/OVN migration. This migration process will run neutron-ovn-db-sync-util with --migrate 3. During the sync we first sync port groups[1] from Neutron DB to OVN DB 4. Then we sync network ports [2]. The process will detect that the network in question is not part of OVN NB. It will create that network in OVN NB db and along with that it will create a metadata port for it(OVN network requires metadataport). The Port_create call will implicitly notify _ensure_default_security_group_handler which will not find securty group for that tenant/project id and create one. Now you have a new security group with 4 new default security group rules. 5. When sync_acls[4] runs it will pick up those 4 new rules but commit to NB DB will fail since the port_group(aka security group) does not exists in NB DB [1] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L104 [2] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L10 [3] https://opendev.org/openstack/neutron/src/branch/master/neutron/db/securitygroups_db.py#L915 [4] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L107 = Ubuntu SRU Details = [Impact] See bug description. [Test Case] Deploy openstack with OVN. Follow steps in "Quick way to reproduce on ML2/OVN" from bug description. [Where problems could occur] The fix mitigates the occurrence of the runtime exception, however the fix retries to sync port groups one more time, so there is potential for the same runtime exception to be raised. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2008943/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2030773] Fix included in openstack/neutron xena-eom
This issue was fixed in the openstack/neutron xena-eom release. ** Changed in: cloud-archive/xena Status: Triaged => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2030773 Title: OVN DB Sync always logs warning messages about updating all router ports Status in Ubuntu Cloud Archive: Fix Released Status in Ubuntu Cloud Archive antelope series: Fix Released Status in Ubuntu Cloud Archive ussuri series: Fix Committed Status in Ubuntu Cloud Archive victoria series: Fix Released Status in Ubuntu Cloud Archive wallaby series: Fix Released Status in Ubuntu Cloud Archive xena series: Fix Released Status in Ubuntu Cloud Archive yoga series: Fix Released Status in Ubuntu Cloud Archive zed series: Fix Released Status in neutron: Fix Released Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Focal: Fix Released Status in neutron source package in Jammy: Fix Released Status in neutron source package in Lunar: Fix Released Bug description: Reported at: https://bugzilla.redhat.com/show_bug.cgi?id=2225156 The ovn-db-sync script does not check if the router ports are actually out-of-sync before adding it to the list of ports that needs to be updated, this can create red herring problems by introducing an irrelevant piece of information [0] in the sync report (specially when ran in "log" mode) making the user think that the databases might be out-of-sync even when it is not. Looking at the code [1] we can see that the comment talks about checking the networks and ipv6_ra_configs changes but it does neither; instead, it adds every router port to the list of ports that needs to be updated. # We dont have to check for the networks and # ipv6_ra_configs values. Lets add it to the # update_lrport_list. If they are in sync, then # update_router_port will be a no-op. update_lrport_list.append(db_router_ports[lrport]) This LP is about changing this behavior and checking for such differences in the router ports before marking them to be updated. [0] 2023-07-24 11:46:31.391 952358 WARNING networking_ovn.ovn_db_sync [req-1081a8a6-82dd-431c-a2ab-f58741dc1677 - - - - -] Router Port port_id=f164c0f1-8ac8-4c45-bba9-8c723a30c701 needs to be updated for networks changed [1] https://github.com/openstack/neutron/blob/c453813d0664259c4da0d132f224be2eebe70072/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L553-L557 = Ubuntu SRU Details = [Impact] See bug description above. [Test Case] Deploy openstack with OVN and multiple routers. Run the ovn-db-sync script and ensure router ports that are not out of sync are not marked to be updated. [Where problems could occur] If the _is_router_port_changed() function had a bug there would be potential for ports to be filtered out that need updating. Presumably this is not the case, but that is a theoritical potential for where problems could occur. All of these patches have already landed in the corresponding upstream branches. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2030773/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2032770] Fix included in openstack/neutron xena-eom
This issue was fixed in the openstack/neutron xena-eom release. ** Changed in: cloud-archive/xena Status: Triaged => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2032770 Title: [SRU] [OVN] port creation with --enable-uplink-status-propagation does not work with OVN mechanism driver Status in Ubuntu Cloud Archive: Fix Released Status in Ubuntu Cloud Archive antelope series: Fix Released Status in Ubuntu Cloud Archive ussuri series: Fix Committed Status in Ubuntu Cloud Archive victoria series: Fix Released Status in Ubuntu Cloud Archive wallaby series: Fix Released Status in Ubuntu Cloud Archive xena series: Fix Released Status in Ubuntu Cloud Archive yoga series: Fix Released Status in Ubuntu Cloud Archive zed series: Fix Released Status in neutron: Fix Released Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Focal: Fix Released Status in neutron source package in Jammy: Fix Released Status in neutron source package in Lunar: Fix Released Bug description: [Impact] This SRU is a backport of https://review.opendev.org/c/openstack/neutron/+/892895 to the respective Ubuntu and UCA releases. The patch is merged to all respective upstream branches (master & stable/[u,v,w,x,y,z,2023.1(a)]). This SRU intends to add the missing 'uplink-status-propagation' extension to ML2/OVN. This extension is already present and working in ML2/OVS, and it is supported by ML2/OVN but the extension is somehow not added to ML2/OVN. The patch simply adds the missing extension to the ML2/OVN too. The impact of this is visible for the deployments migrating from ML2/OVS to ML2/OVN. The following command fails to work on ML2/OVN: ``` openstack port create --network 8d30fb08-2c6a-42fd-98c4-223d345c8c4f --binding-profile trusted=true --enable-uplink-status-propagation --vnic-type direct aaa # BadRequestException: 400: Client Error for url: https://mycloud.example.com:9696/v2.0/ports, Unrecognized attribute(s) 'propagate_uplink_status' ``` The fix corrects this behavior by adding the missing extension. [Test Case] - Deploy a Focal/Yoga cloud: - ./generate-bundle.sh -s focal -r yoga --name test-focal-yoga-stack --run --ovn # After the dust settles - ./configure - source ./novarc - openstack port create --network --binding-profile trusted=true --enable-uplink-status-propagation --vnic-type direct aaa - It should fail with "BadRequestException: 400: Client Error for url: https://mycloud.example.com:9696/v2.0/ports, Unrecognized attribute(s) 'propagate_uplink_status'" To confirm the fix, repeat the scenario and observe that the error disappears and port creation succeeds. [Regression Potential] The patch is quite trivial and should not affect any deployment negatively. The extension is optional and disabled by default. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2032770/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2055886] Re: [FT] Check port binding object in "test_virtual_port_host_update_upon_failover"
Reviewed: https://review.opendev.org/c/openstack/neutron/+/910941 Committed: https://opendev.org/openstack/neutron/commit/8b007e63664ef811671554b7c878cb552078aa99 Submitter: "Zuul (22348)" Branch:master commit 8b007e63664ef811671554b7c878cb552078aa99 Author: Rodolfo Alonso Hernandez Date: Mon Mar 4 14:00:07 2024 + [FT] Check "Port_Binding" register exists before checking type In "test_virtual_port_host_update_upon_failover", it is needed to check if the "Port_Binding" register exists before checking the type. Closes-Bug: #2055886 Change-Id: I8a6b3498803bcba592a82dfbe43a39137dd12fa2 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2055886 Title: [FT] Check port binding object in "test_virtual_port_host_update_upon_failover" Status in neutron: Fix Released Bug description: Error: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_44f/898238/27/check/neutron- functional-with-uwsgi/44f78d4/testr_results.html Snippet: https://paste.opendev.org/show/bDneDxmXx4AiBscSfrbE/ The port binding check failed because the "Port_Binding" register was not created yet. We should consider that option when checking that port binding type. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2055886/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2008943] Fix included in openstack/neutron wallaby-eom
This issue was fixed in the openstack/neutron wallaby-eom release. ** Changed in: cloud-archive/wallaby Status: Triaged => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2008943 Title: OVN DB Sync utility cannot find NB DB Port Group Status in Ubuntu Cloud Archive: Fix Released Status in Ubuntu Cloud Archive ussuri series: Fix Committed Status in Ubuntu Cloud Archive victoria series: Fix Released Status in Ubuntu Cloud Archive wallaby series: Fix Released Status in Ubuntu Cloud Archive xena series: Fix Released Status in neutron: In Progress Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Focal: Fix Released Bug description: Runtime exception: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_aa9f203b_ec51_4893_9bda_cfadbff9f800 can occure while performing database sync between Neutron db and OVN NB db using neutron-ovn-db-sync-util. This exception occures when the `sync_networks_ports_and_dhcp_opts()` function ends up implicitly creating a new default security group for a tenant/project id. This is normally ok but the problem is that `sync_port_groups` was already called and thus the port_group does not exists in NB DB. When the `sync_acls()` is called later there is no port group found and exception occurs. Quick way to reproduce on ML2/OVN: - openstack project create test_project - openstack create network --project test_project test_network - openstack port delete $(openstack port list --network test_network -c ID -f value) # since this is an empty network only the metadata port should get listed and subsequently deleted - openstack security group delete test_project So now that you have a network without a metadata port in it and no default security group for the project/tenant that this network belongs to run neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn- neutron_sync_mode migrate The exeption should occur Here is a more realistic scenario how we can run into this with ML2/OVS to ML2/OVN migration. I am also including why the code runs into it. 1. ML2/OVS enviroment with a network but no default security group for the project/tenant associated with the network 2. Perform ML2/OVS to ML2/OVN migration. This migration process will run neutron-ovn-db-sync-util with --migrate 3. During the sync we first sync port groups[1] from Neutron DB to OVN DB 4. Then we sync network ports [2]. The process will detect that the network in question is not part of OVN NB. It will create that network in OVN NB db and along with that it will create a metadata port for it(OVN network requires metadataport). The Port_create call will implicitly notify _ensure_default_security_group_handler which will not find securty group for that tenant/project id and create one. Now you have a new security group with 4 new default security group rules. 5. When sync_acls[4] runs it will pick up those 4 new rules but commit to NB DB will fail since the port_group(aka security group) does not exists in NB DB [1] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L104 [2] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L10 [3] https://opendev.org/openstack/neutron/src/branch/master/neutron/db/securitygroups_db.py#L915 [4] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L107 = Ubuntu SRU Details = [Impact] See bug description. [Test Case] Deploy openstack with OVN. Follow steps in "Quick way to reproduce on ML2/OVN" from bug description. [Where problems could occur] The fix mitigates the occurrence of the runtime exception, however the fix retries to sync port groups one more time, so there is potential for the same runtime exception to be raised. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2008943/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2030773] Fix included in openstack/neutron wallaby-eom
This issue was fixed in the openstack/neutron wallaby-eom release. ** Changed in: cloud-archive/wallaby Status: Triaged => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2030773 Title: OVN DB Sync always logs warning messages about updating all router ports Status in Ubuntu Cloud Archive: Fix Released Status in Ubuntu Cloud Archive antelope series: Fix Released Status in Ubuntu Cloud Archive ussuri series: Fix Committed Status in Ubuntu Cloud Archive victoria series: Fix Released Status in Ubuntu Cloud Archive wallaby series: Fix Released Status in Ubuntu Cloud Archive xena series: Fix Released Status in Ubuntu Cloud Archive yoga series: Fix Released Status in Ubuntu Cloud Archive zed series: Fix Released Status in neutron: Fix Released Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Focal: Fix Released Status in neutron source package in Jammy: Fix Released Status in neutron source package in Lunar: Fix Released Bug description: Reported at: https://bugzilla.redhat.com/show_bug.cgi?id=2225156 The ovn-db-sync script does not check if the router ports are actually out-of-sync before adding it to the list of ports that needs to be updated, this can create red herring problems by introducing an irrelevant piece of information [0] in the sync report (specially when ran in "log" mode) making the user think that the databases might be out-of-sync even when it is not. Looking at the code [1] we can see that the comment talks about checking the networks and ipv6_ra_configs changes but it does neither; instead, it adds every router port to the list of ports that needs to be updated. # We dont have to check for the networks and # ipv6_ra_configs values. Lets add it to the # update_lrport_list. If they are in sync, then # update_router_port will be a no-op. update_lrport_list.append(db_router_ports[lrport]) This LP is about changing this behavior and checking for such differences in the router ports before marking them to be updated. [0] 2023-07-24 11:46:31.391 952358 WARNING networking_ovn.ovn_db_sync [req-1081a8a6-82dd-431c-a2ab-f58741dc1677 - - - - -] Router Port port_id=f164c0f1-8ac8-4c45-bba9-8c723a30c701 needs to be updated for networks changed [1] https://github.com/openstack/neutron/blob/c453813d0664259c4da0d132f224be2eebe70072/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L553-L557 = Ubuntu SRU Details = [Impact] See bug description above. [Test Case] Deploy openstack with OVN and multiple routers. Run the ovn-db-sync script and ensure router ports that are not out of sync are not marked to be updated. [Where problems could occur] If the _is_router_port_changed() function had a bug there would be potential for ports to be filtered out that need updating. Presumably this is not the case, but that is a theoritical potential for where problems could occur. All of these patches have already landed in the corresponding upstream branches. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2030773/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2032770] Fix included in openstack/neutron wallaby-eom
This issue was fixed in the openstack/neutron wallaby-eom release. ** Changed in: cloud-archive/wallaby Status: Triaged => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2032770 Title: [SRU] [OVN] port creation with --enable-uplink-status-propagation does not work with OVN mechanism driver Status in Ubuntu Cloud Archive: Fix Released Status in Ubuntu Cloud Archive antelope series: Fix Released Status in Ubuntu Cloud Archive ussuri series: Fix Committed Status in Ubuntu Cloud Archive victoria series: Fix Released Status in Ubuntu Cloud Archive wallaby series: Fix Released Status in Ubuntu Cloud Archive xena series: Fix Released Status in Ubuntu Cloud Archive yoga series: Fix Released Status in Ubuntu Cloud Archive zed series: Fix Released Status in neutron: Fix Released Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Focal: Fix Released Status in neutron source package in Jammy: Fix Released Status in neutron source package in Lunar: Fix Released Bug description: [Impact] This SRU is a backport of https://review.opendev.org/c/openstack/neutron/+/892895 to the respective Ubuntu and UCA releases. The patch is merged to all respective upstream branches (master & stable/[u,v,w,x,y,z,2023.1(a)]). This SRU intends to add the missing 'uplink-status-propagation' extension to ML2/OVN. This extension is already present and working in ML2/OVS, and it is supported by ML2/OVN but the extension is somehow not added to ML2/OVN. The patch simply adds the missing extension to the ML2/OVN too. The impact of this is visible for the deployments migrating from ML2/OVS to ML2/OVN. The following command fails to work on ML2/OVN: ``` openstack port create --network 8d30fb08-2c6a-42fd-98c4-223d345c8c4f --binding-profile trusted=true --enable-uplink-status-propagation --vnic-type direct aaa # BadRequestException: 400: Client Error for url: https://mycloud.example.com:9696/v2.0/ports, Unrecognized attribute(s) 'propagate_uplink_status' ``` The fix corrects this behavior by adding the missing extension. [Test Case] - Deploy a Focal/Yoga cloud: - ./generate-bundle.sh -s focal -r yoga --name test-focal-yoga-stack --run --ovn # After the dust settles - ./configure - source ./novarc - openstack port create --network --binding-profile trusted=true --enable-uplink-status-propagation --vnic-type direct aaa - It should fail with "BadRequestException: 400: Client Error for url: https://mycloud.example.com:9696/v2.0/ports, Unrecognized attribute(s) 'propagate_uplink_status'" To confirm the fix, repeat the scenario and observe that the error disappears and port creation succeeds. [Regression Potential] The patch is quite trivial and should not affect any deployment negatively. The extension is optional and disabled by default. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2032770/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1955578] Fix included in openstack/neutron victoria-eom
This issue was fixed in the openstack/neutron victoria-eom release.
** Changed in: cloud-archive/victoria
Status: Triaged => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1955578
Title:
OVN transaction could not be completed due to a race condition
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive ussuri series:
Fix Committed
Status in Ubuntu Cloud Archive victoria series:
Fix Released
Status in neutron:
Fix Released
Status in neutron package in Ubuntu:
Fix Released
Status in neutron source package in Focal:
Fix Released
Bug description:
When executing the test "test_connectivity_through_2_routers" it is highly
possible to have
a race condition:
networking_ovn.common.exceptions.RevisionConflict: OVN revision number
for {PORT_ID} (type: ports) is equal or higher than the given
resource. Skipping update.
Bugzilla reference:
https://bugzilla.redhat.com/show_bug.cgi?id=1860448
= Ubuntu SRU Details =
[Impact]
See bug description.
[Test Case]
Deploy openstack with OVN. Run the test_connectivity_through_2_routers test
from https://github.com/openstack/neutron-tempest-plugin. This could also be
tested manually based on what that test does. Ensure the router port status is
not set to DOWN at any point.
[Where problems could occur]
The existing bug could still occur if the assumpion that specifying the port
type is not correct. Presumably this is not the case, but that is a theoritical
potential for where problems could occur. All of these patches have already
landed in the corresponding upstream branches.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1955578/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2008943] Fix included in openstack/neutron victoria-eom
This issue was fixed in the openstack/neutron victoria-eom release. ** Changed in: cloud-archive/victoria Status: Triaged => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2008943 Title: OVN DB Sync utility cannot find NB DB Port Group Status in Ubuntu Cloud Archive: Fix Released Status in Ubuntu Cloud Archive ussuri series: Fix Committed Status in Ubuntu Cloud Archive victoria series: Fix Released Status in Ubuntu Cloud Archive wallaby series: Triaged Status in Ubuntu Cloud Archive xena series: Triaged Status in neutron: In Progress Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Focal: Fix Released Bug description: Runtime exception: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_aa9f203b_ec51_4893_9bda_cfadbff9f800 can occure while performing database sync between Neutron db and OVN NB db using neutron-ovn-db-sync-util. This exception occures when the `sync_networks_ports_and_dhcp_opts()` function ends up implicitly creating a new default security group for a tenant/project id. This is normally ok but the problem is that `sync_port_groups` was already called and thus the port_group does not exists in NB DB. When the `sync_acls()` is called later there is no port group found and exception occurs. Quick way to reproduce on ML2/OVN: - openstack project create test_project - openstack create network --project test_project test_network - openstack port delete $(openstack port list --network test_network -c ID -f value) # since this is an empty network only the metadata port should get listed and subsequently deleted - openstack security group delete test_project So now that you have a network without a metadata port in it and no default security group for the project/tenant that this network belongs to run neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn- neutron_sync_mode migrate The exeption should occur Here is a more realistic scenario how we can run into this with ML2/OVS to ML2/OVN migration. I am also including why the code runs into it. 1. ML2/OVS enviroment with a network but no default security group for the project/tenant associated with the network 2. Perform ML2/OVS to ML2/OVN migration. This migration process will run neutron-ovn-db-sync-util with --migrate 3. During the sync we first sync port groups[1] from Neutron DB to OVN DB 4. Then we sync network ports [2]. The process will detect that the network in question is not part of OVN NB. It will create that network in OVN NB db and along with that it will create a metadata port for it(OVN network requires metadataport). The Port_create call will implicitly notify _ensure_default_security_group_handler which will not find securty group for that tenant/project id and create one. Now you have a new security group with 4 new default security group rules. 5. When sync_acls[4] runs it will pick up those 4 new rules but commit to NB DB will fail since the port_group(aka security group) does not exists in NB DB [1] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L104 [2] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L10 [3] https://opendev.org/openstack/neutron/src/branch/master/neutron/db/securitygroups_db.py#L915 [4] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L107 = Ubuntu SRU Details = [Impact] See bug description. [Test Case] Deploy openstack with OVN. Follow steps in "Quick way to reproduce on ML2/OVN" from bug description. [Where problems could occur] The fix mitigates the occurrence of the runtime exception, however the fix retries to sync port groups one more time, so there is potential for the same runtime exception to be raised. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2008943/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2030773] Fix included in openstack/neutron victoria-eom
This issue was fixed in the openstack/neutron victoria-eom release. ** Changed in: cloud-archive/victoria Status: Triaged => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2030773 Title: OVN DB Sync always logs warning messages about updating all router ports Status in Ubuntu Cloud Archive: Fix Released Status in Ubuntu Cloud Archive antelope series: Fix Released Status in Ubuntu Cloud Archive ussuri series: Fix Committed Status in Ubuntu Cloud Archive victoria series: Fix Released Status in Ubuntu Cloud Archive wallaby series: Triaged Status in Ubuntu Cloud Archive xena series: Triaged Status in Ubuntu Cloud Archive yoga series: Fix Released Status in Ubuntu Cloud Archive zed series: Fix Released Status in neutron: Fix Released Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Focal: Fix Released Status in neutron source package in Jammy: Fix Released Status in neutron source package in Lunar: Fix Released Bug description: Reported at: https://bugzilla.redhat.com/show_bug.cgi?id=2225156 The ovn-db-sync script does not check if the router ports are actually out-of-sync before adding it to the list of ports that needs to be updated, this can create red herring problems by introducing an irrelevant piece of information [0] in the sync report (specially when ran in "log" mode) making the user think that the databases might be out-of-sync even when it is not. Looking at the code [1] we can see that the comment talks about checking the networks and ipv6_ra_configs changes but it does neither; instead, it adds every router port to the list of ports that needs to be updated. # We dont have to check for the networks and # ipv6_ra_configs values. Lets add it to the # update_lrport_list. If they are in sync, then # update_router_port will be a no-op. update_lrport_list.append(db_router_ports[lrport]) This LP is about changing this behavior and checking for such differences in the router ports before marking them to be updated. [0] 2023-07-24 11:46:31.391 952358 WARNING networking_ovn.ovn_db_sync [req-1081a8a6-82dd-431c-a2ab-f58741dc1677 - - - - -] Router Port port_id=f164c0f1-8ac8-4c45-bba9-8c723a30c701 needs to be updated for networks changed [1] https://github.com/openstack/neutron/blob/c453813d0664259c4da0d132f224be2eebe70072/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L553-L557 = Ubuntu SRU Details = [Impact] See bug description above. [Test Case] Deploy openstack with OVN and multiple routers. Run the ovn-db-sync script and ensure router ports that are not out of sync are not marked to be updated. [Where problems could occur] If the _is_router_port_changed() function had a bug there would be potential for ports to be filtered out that need updating. Presumably this is not the case, but that is a theoritical potential for where problems could occur. All of these patches have already landed in the corresponding upstream branches. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2030773/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2032770] Fix included in openstack/neutron victoria-eom
This issue was fixed in the openstack/neutron victoria-eom release. ** Changed in: cloud-archive/victoria Status: Triaged => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2032770 Title: [SRU] [OVN] port creation with --enable-uplink-status-propagation does not work with OVN mechanism driver Status in Ubuntu Cloud Archive: Fix Released Status in Ubuntu Cloud Archive antelope series: Fix Released Status in Ubuntu Cloud Archive ussuri series: Fix Committed Status in Ubuntu Cloud Archive victoria series: Fix Released Status in Ubuntu Cloud Archive wallaby series: Triaged Status in Ubuntu Cloud Archive xena series: Triaged Status in Ubuntu Cloud Archive yoga series: Fix Released Status in Ubuntu Cloud Archive zed series: Fix Released Status in neutron: Fix Released Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Focal: Fix Released Status in neutron source package in Jammy: Fix Released Status in neutron source package in Lunar: Fix Released Bug description: [Impact] This SRU is a backport of https://review.opendev.org/c/openstack/neutron/+/892895 to the respective Ubuntu and UCA releases. The patch is merged to all respective upstream branches (master & stable/[u,v,w,x,y,z,2023.1(a)]). This SRU intends to add the missing 'uplink-status-propagation' extension to ML2/OVN. This extension is already present and working in ML2/OVS, and it is supported by ML2/OVN but the extension is somehow not added to ML2/OVN. The patch simply adds the missing extension to the ML2/OVN too. The impact of this is visible for the deployments migrating from ML2/OVS to ML2/OVN. The following command fails to work on ML2/OVN: ``` openstack port create --network 8d30fb08-2c6a-42fd-98c4-223d345c8c4f --binding-profile trusted=true --enable-uplink-status-propagation --vnic-type direct aaa # BadRequestException: 400: Client Error for url: https://mycloud.example.com:9696/v2.0/ports, Unrecognized attribute(s) 'propagate_uplink_status' ``` The fix corrects this behavior by adding the missing extension. [Test Case] - Deploy a Focal/Yoga cloud: - ./generate-bundle.sh -s focal -r yoga --name test-focal-yoga-stack --run --ovn # After the dust settles - ./configure - source ./novarc - openstack port create --network --binding-profile trusted=true --enable-uplink-status-propagation --vnic-type direct aaa - It should fail with "BadRequestException: 400: Client Error for url: https://mycloud.example.com:9696/v2.0/ports, Unrecognized attribute(s) 'propagate_uplink_status'" To confirm the fix, repeat the scenario and observe that the error disappears and port creation succeeds. [Regression Potential] The patch is quite trivial and should not affect any deployment negatively. The extension is optional and disabled by default. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2032770/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2050236] Re: Automatic L3 Agent Failover does not work for HA Routers
Reviewed: https://review.opendev.org/c/openstack/neutron/+/908434 Committed: https://opendev.org/openstack/neutron/commit/fa28c3c35cc7ae5b25441d3d83bf78f00a54f477 Submitter: "Zuul (22348)" Branch:master commit fa28c3c35cc7ae5b25441d3d83bf78f00a54f477 Author: Christian Rohmann Date: Thu Feb 8 15:27:18 2024 +0100 Allow HA routers to have automatic l3agent failover Currently routers with ha=true are NOT rescheduled form a dead L3 agent, even when `allow_automatic_l3agent_failover` is enabled. This is contrary to what the user expects and the feature description states: "Automatically reschedule routers from offline L3 agents to online L3 agents." There is no distinction made between HA and non-HA routers. Also HA and automatic-failover can work together: * HA allows for a fast failover to a standby router * Automatic failover then restores back full redundancy in case the failed L3 agent, which HA failed away from, does not come back within a certain time. Closes-Bug: #2050236 Change-Id: I1e5ee5048f61eef7fa4d9de25e69bf0e0a5ea442 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2050236 Title: Automatic L3 Agent Failover does not work for HA Routers Status in neutron: Fix Released Bug description: When enabling allow_automatic_l3agent_failover in the neutron config, neutron does not automatically reschedule routers with the HA parameter. In https://github.com/openstack/neutron/blob/master/neutron/db/agentschedulers_db.py#L131, neutron loads all downed agents into the down_bindings list and then iterates over it. This list gets populated in this method: https://github.com/openstack/neutron/blob/master/neutron/objects/l3agent.py#L55 Here, a SQL query gets generated that specifically excludes routers that have the HA-parameter (l3_attrs.RouterExtraAttributes.ha) set to something other than false or NULL. This behavior has always been the case and got introduced in https://github.com/openstack/neutron/commit/7e51f2aea517e1431b9a860c45761c057710f5b2#diff-c1382d923846002c27484cca55e555689da1e08c22f1f8a0343726fd881e86ce. Is there a specific reason for why HA routers get excluded? If not, this conditional in the SQL statement can be removed. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2050236/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2056300] [NEW] Some Roboto fonts are looked in the wrong location
Public bug reported:
I do have a basic OpenStack-Ansible All-in-One deployment of the current
master branch.
Horizon SHA: a77818d745f25ed1e5b576760d3481f0f62ce042
Deployment was done following way:
1. git clone https://opendev.org/openstack/openstack-ansible
2. cd openstack-ansible
3. ./scripts/gate-check-commit.sh aio_lxc
Once it's done, horizon is working properly. However, from time to time
I do catch "429 Too many requests" which is potentially legit behavior
by LB configuration, that tries to implement bot protection:
http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 }:
https://opendev.org/openstack/openstack-ansible/src/commit/e72984ca956d44d8056ca4ea8ea7e263bc3c8881/inventory/group_vars/horizon_all/haproxy_service.yml#L24
Violation of this rule is triggered by invalid path to roboto fontface
which returns quite some 404 per each request:
[06/Mar/2024:08:58:13.851] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/1/1/2 404 340 - -
32/1/0/0/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/Roboto/Roboto-Regular.woff2
HTTP/2.0"
[06/Mar/2024:08:58:13.856] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/0/1/1 404 340 - -
32/1/0/0/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/Roboto/Roboto-Bold.woff2
HTTP/2.0"
On the horizon container this path is indeed not valid:
root@aio1-horizon-container-f7cace3d:/# ls -l
/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-packages/static/horizon/lib/roboto_fontface/fonts/Roboto/Roboto-Bold.woff2ls:
cannot access
'/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-packages/static/horizon/lib/roboto_fontface/fonts/Roboto/Roboto-Bold.woff2':
No such file or directory
root@aio1-horizon-container-f7cace3d:/#
However, expected path is slightly different:
root@aio1-horizon-container-f7cace3d:/# ls -l
/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-packages/static/horizon/lib/roboto_fontface/fonts/roboto/Roboto-Bold.woff2
-rwxr-xr-x 1 root root 63596 Mar 6 08:37
/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-packages/static/horizon/lib/roboto_fontface/fonts/roboto/Roboto-Bold.woff2
root@aio1-horizon-container-f7cace3d:/#
So doing smth like `cp -r
/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-
packages/static/horizon/lib/roboto_fontface/fonts/roboto/
/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-
packages/static/horizon/lib/roboto_fontface/fonts/Roboto/` does the
trick.
And the issue is basically in `roboto` vs `Roboto` in path.
If I would try to `mv roboto Roboto` - then some dashboard will fail,
like heat-dashboard, as they do expect `roboto`:
[06/Mar/2024:09:10:30.452] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/0/0/0 404 340 - -
38/1/2/2/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/roboto/Roboto-Regular.woff2
HTTP/2.0"
[06/Mar/2024:09:10:30.560] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/0/3/3 404 340 - -
38/1/2/2/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/roboto/Roboto-Regular.woff
HTTP/2.0"
[06/Mar/2024:09:10:30.604] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/0/1/1 404 340 - -
38/1/2/2/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/roboto/Roboto-Regular.ttf
HTTP/2.0"
The /roboto directory is being populated automatically during `horizon-
manage.py collectstatic --noinput` execution.
** Affects: horizon
Importance: Undecided
Status: New
** Description changed:
I do have a basic OpenStack-Ansible All-in-One deployment of the current
master branch.
Horizon SHA: a77818d745f25ed1e5b576760d3481f0f62ce042
Deployment was done following way:
1. git clone https://opendev.org/openstack/openstack-ansible
2. cd openstack-ansible
3. ./scripts/gate-check-commit.sh aio_lxc
Once it's done, horizon is working properly. However, from time to time
I do catch "429 Too many requests" which is potentially legit behavior
by LB configuration, that tries to implement bot protection:
http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 }:
https://opendev.org/openstack/openstack-ansible/src/commit/e72984ca956d44d8056ca4ea8ea7e263bc3c8881/inventory/group_vars/horizon_all/haproxy_service.yml#L24
-
- Violation of this rule is triggered by invalid path to roboto fontface which
returns quite some 404 per each request:
+ Violation of this rule is triggered by invalid path to roboto fontface
+ which returns quite some 404 per each request:
[06/Mar/2024:08:58:13.851] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/1/1/2 404 340 - -
32/1/0/0/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/Roboto/Roboto-Regular.woff2
HTTP/2.0"
[06/Mar/2024:08:58:13.856] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/0/1/1 404
