[Yahoo-eng-team] [Bug 1918397] [NEW] failed to name sriov vfs because the names reserved for representors
Public bug reported: After providing a patch to bind sriov vfs in mlnx devices after moving to switchdec, we saw in some cases that the names of the representors were reserved before for the vfs and that failed to get the old names again of the vfs after binding them and name them as ethX. This creates lot of confusion because now all VFs have 2 visible netdevices, one being the VF representor, and the other being a mlx5-bound VF ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1918397 Title: failed to name sriov vfs because the names reserved for representors Status in neutron: New Bug description: After providing a patch to bind sriov vfs in mlnx devices after moving to switchdec, we saw in some cases that the names of the representors were reserved before for the vfs and that failed to get the old names again of the vfs after binding them and name them as ethX. This creates lot of confusion because now all VFs have 2 visible netdevices, one being the VF representor, and the other being a mlx5-bound VF To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1918397/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1915308] Re: security group table doesn't observe Neutron policy settings
It is fixed by https://review.opendev.org/c/openstack/horizon/+/774922 ** Changed in: horizon Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1915308 Title: security group table doesn't observe Neutron policy settings Status in OpenStack Dashboard (Horizon): Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: The security group panel enables all actions (create/edit/delete/manage rules/etc.) regardless of the network policy.yaml settings or user account. In the code there's this telling readme: # TODO(amotoki): [drop-nova-network] Add neutron policy support In my deployment this is a bit alarming -- users who are intended to be read-only are nonetheless invited to delete things. Of course the Neutron backend /does/ observe the policy so this is ugly but not usually an actual security risk unless people have different back-end and front-end policy files. I'm flagging as security-related nonetheless for the odd edge case where it poses a risk. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1915308/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1793411] Re: Dashboard memory leaks
As mentioned in the above comments it is already fixed by https://review.openstack.org/c/614893/. So changing bug status to Fix Released and regarding the same issue in Queen and Rocky branch, you can apply the same fix in your env. I don't have an evn. to check if above patch not work in the old branches. Last but not least Rocky and Queen are EOL, So I am not sure it is good to purpose backport to old branches now. Feel Free to add your suggestions. ** Changed in: horizon Status: Fix Committed => Fix Released ** Changed in: horizon Importance: High => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1793411 Title: Dashboard memory leaks Status in OpenStack Dashboard (Horizon): Fix Released Bug description: 1.Issue description Recently, we found the server which hosts horizon dashboard had serveral times OOM caused by horizon services. After restarting the dashboard, the memory usage goes up very quickly if we access /project/network_topology/ path. 2.How to reproduce Login into the dashboard and go to 'Network Topology' tab, then leave it there (autorefresh 10s by default), now monitor the memory changes on the host. 3.Versions and Components Dashboard: Stable/Pike Server: uWSGI 1.9.17-1 OS: Ubuntu 14.04 trusty Python: 2.7.6 As the codes of memoized has little changes since Pike, if you use Queen/Rocky release, you may also succeed to reproduce it. 4.The investigation The root cause of the memory leak is the decorator memorized(horizon/utils/memoized.py) which is used to cache function calls in Horizon. After disable it, the memory increases has been controlled. The following is the comparison of memory change(with guppy) for each request of /project/network_topology: - original (no code change)684kb - do garbage collection manually 185kb - disable memorize cache 10kb As we known, memoized uses weakref to cache objects. A weak reference to an object is not enough to keep the object alive: when the only remaining references to a referent are weak references, garbage collection is free to destroy the referent and reuse its memory for something else. In the memory, we could see lots of weakref stuffs, the following is a example: Partition of a set of 394 objects. Total size = 37824 bytes. Index Count % Size % Cumulative % Kind (class / dict of class) 0197 5018912 50 18912 50 _cffi_backend.CDataGCP 1197 5018912 50 37824 100 weakref.KeyedRefq But the rest of them are not. the following result is the memory objects changes of per /project/network_topology access with garbage collection manually. Partition of a set of 1017 objects. Total size = 183680 bytes. Index Count % Size % Cumulative % Referrers by Kind (class / dict of class) 0419 4158320 32 58320 32 dict (no owner) 1100 1023416 13 81736 44 list 2135 1315184 8 96920 53 3 2 0 6704 4103624 56 urllib3.connection.VerifiedHTTPSConnection 4 2 0 6704 4110328 60 urllib3.connectionpool.HTTPSConnectionPool 5 1 0 3352 2113680 62 novaclient.v2.client.Client 6 2 0 2096 1115776 63 OpenSSL.SSL.Connection 7 2 0 2096 1117872 64 OpenSSL.SSL.Context 8 2 0 2096 1119968 65 Queue.LifoQueue 9 12 1 2096 1122064 66 dict of urllib3.connectionpool.HTTPSConnectionPool The most of them are dicts. Followings are the dicts sorted by class, as you can see most of them are not weakref objects: Partition of a set of 419 objects. Total size = 58320 bytes. Index Count % Size % Cumulative % Class 0362 8650712 87 50712 87 unicode 1 27 6 3736 6 54448 93 list 2 5 1 2168 4 56616 97 dict 3 22 5 1448 2 58064 100 str 4 2 0 192 0 58256 100 weakref.KeyedRef 5 1 0 64 0 58320 100 keystoneauth1.discover.Discover 5.The issue So the problem is that memoized does not work like what we expect. It allocates memory to cache objects but some of them could not be released. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1793411/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918352] [NEW] Fault Injection #2065 - improve unit test effectiveness
Public bug reported: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WVIV (Wrong value used in variable initialization) is a type of fault where the value used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code|Original Code |Incorrect Code 5515|is_uuid = False|is_uuid = True Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918352 Title: Fault Injection #2065 - improve unit test effectiveness Status in OpenStack Compute (nova): New Bug description: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WVIV (Wrong value used in variable initialization) is a type of fault where the value used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code |Original Code |Incorrect Code 5515 |is_uuid = False|is_uuid = True Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918352/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918350] [NEW] Fault Injection #2045 - improve unit test effectiveness
Public bug reported: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WVIV (Wrong value used in variable initialization) is a type of fault where the value used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code|Original Code |Incorrect Code 1251|req_spec.is_bfv = False|req_spec.is_bfv = True Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918350 Title: Fault Injection #2045 - improve unit test effectiveness Status in OpenStack Compute (nova): New Bug description: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WVIV (Wrong value used in variable initialization) is a type of fault where the value used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code |Original Code |Incorrect Code 1251 |req_spec.is_bfv = False|req_spec.is_bfv = True Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918350/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918348] [NEW] Fault Injection #2041 - improve unit test effectiveness
Public bug reported: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WVIV (Wrong value used in variable initialization) is a type of fault where the value used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code|Original Code |Incorrect Code 108 |MIN_COMPUTE_SAME_HOST_COLD_MIGRATE = 48 |MIN_COMPUTE_SAME_HOST_COLD_MIGRATE = 655654 Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918348 Title: Fault Injection #2041 - improve unit test effectiveness Status in OpenStack Compute (nova): New Bug description: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WVIV (Wrong value used in variable initialization) is a type of fault where the value used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code |Original Code |Incorrect Code 108 |MIN_COMPUTE_SAME_HOST_COLD_MIGRATE = 48 |MIN_COMPUTE_SAME_HOST_COLD_MIGRATE = 655654 Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918348/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918347] [NEW] Fault Injection #2039 - improve unit test effectiveness
Public bug reported: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WVIV (Wrong value used in variable initialization) is a type of fault where the value used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code|Original Code |Incorrect Code 106 |MIN_COMPUTE_SYNC_COMPUTE_STATUS_DISABLED = 38 |MIN_COMPUTE_SYNC_COMPUTE_STATUS_DISABLED = 56484 Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918347 Title: Fault Injection #2039 - improve unit test effectiveness Status in OpenStack Compute (nova): New Bug description: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WVIV (Wrong value used in variable initialization) is a type of fault where the value used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code |Original Code |Incorrect Code 106 |MIN_COMPUTE_SYNC_COMPUTE_STATUS_DISABLED = 38 |MIN_COMPUTE_SYNC_COMPUTE_STATUS_DISABLED = 56484 Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918347/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918349] [NEW] Fault Injection #2042 - improve unit test effectiveness
Public bug reported: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WVIV (Wrong value used in variable initialization) is a type of fault where the value used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code|Original Code |Incorrect Code 107 |MIN_COMPUTE_CROSS_CELL_RESIZE = 47 |MIN_COMPUTE_CROSS_CELL_RESIZE = 47546 Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918349 Title: Fault Injection #2042 - improve unit test effectiveness Status in OpenStack Compute (nova): New Bug description: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WVIV (Wrong value used in variable initialization) is a type of fault where the value used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code |Original Code |Incorrect Code 107 |MIN_COMPUTE_CROSS_CELL_RESIZE = 47 |MIN_COMPUTE_CROSS_CELL_RESIZE = 47546 Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918349/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918346] [NEW] Fault Injection #7 - improve unit test effectiveness
Public bug reported: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code|Original Code |Incorrect Code 3152|attr = 'task_state'|attr = 'K53RIP5A' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918346 Title: Fault Injection #7 - improve unit test effectiveness Status in OpenStack Compute (nova): New Bug description: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code |Original Code |Incorrect Code 3152 |attr = 'task_state'|attr = 'K53RIP5A' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918346/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918344] [NEW] Fault Injection #4 - improve unit test effectiveness
Public bug reported: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of CodeOriginal Code Incorrect Code 104 AGGREGATE_ACTION_ADD = 'Add' AGGREGATE_ACTION_ADD = '71ENJL8W' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918344 Title: Fault Injection #4 - improve unit test effectiveness Status in OpenStack Compute (nova): New Bug description: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code Original Code Incorrect Code 104 AGGREGATE_ACTION_ADD = 'Add' AGGREGATE_ACTION_ADD = '71ENJL8W' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918344/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918345] [NEW] Fault Injection #6 - improve unit test effectiveness
Public bug reported: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code|Original Code |Incorrect Code 3155|attr = 'vm_state' |attr = 'KMJHW8MG' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918345 Title: Fault Injection #6 - improve unit test effectiveness Status in OpenStack Compute (nova): New Bug description: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code |Original Code |Incorrect Code 3155 |attr = 'vm_state' |attr = 'KMJHW8MG' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918345/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918340] [NEW] Fault Injection #1 - improve unit test effectiveness
Public bug reported: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of CodeOriginal Code Incorrect Code 102 AGGREGATE_ACTION_UPDATE_META = 'UpdateMeta' AGGREGATE_ACTION_UPDATE_META = 'NHZWTCGB' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918340 Title: Fault Injection #1 - improve unit test effectiveness Status in OpenStack Compute (nova): New Bug description: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code Original Code Incorrect Code 102 AGGREGATE_ACTION_UPDATE_META = 'UpdateMeta' AGGREGATE_ACTION_UPDATE_META = 'NHZWTCGB' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918340/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918342] [NEW] Fault Injection #2 - improve unit test effectiveness
Public bug reported: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of CodeOriginal Code Incorrect Code 101 AGGREGATE_ACTION_UPDATE = 'Update' AGGREGATE_ACTION_UPDATE = 'BGIAPGHU' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918342 Title: Fault Injection #2 - improve unit test effectiveness Status in OpenStack Compute (nova): New Bug description: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code Original Code Incorrect Code 101 AGGREGATE_ACTION_UPDATE = 'Update' AGGREGATE_ACTION_UPDATE = 'BGIAPGHU' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918342/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918343] [NEW] Fault Injection #3 - improve unit test effectiveness
Public bug reported: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of CodeOriginal Code Incorrect Code 103 AGGREGATE_ACTION_DELETE = 'Delete' AGGREGATE_ACTION_DELETE = '3E6KOPNF' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918343 Title: Fault Injection #3 - improve unit test effectiveness Status in OpenStack Compute (nova): New Bug description: Description === I have performed fault injection in openstack nova by changing the code of compute/api.py (inserting a representative/probable bug) and then ran the unit, functional and integration tests and discover that some of the bugs inserted were not detected by the test suite: The reference WIDS (Wrong string in initial data) is a type of fault where the string used in a variable initialization is set to an incorrect value. Steps to reproduce == Line of Code Original Code Incorrect Code 103 AGGREGATE_ACTION_DELETE = 'Delete' AGGREGATE_ACTION_DELETE = '3E6KOPNF' Refactor the line of code above to the incorrect code. Then execute the unit tests. Expected result === The unit tests should detect the fault. Actual result === The fault was not detected by the unit tests. Environment === The code tested is on the stable/ussuri branch. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918343/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1896734] Re: A privsep daemon spawned by neutron-openvswitch-agent hangs when debug logging is enabled (large number of registered NICs) - an RPC response is too large for msgpac
** Also affects: neutron (Ubuntu) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: python-oslo.privsep (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Hirsute) Importance: Undecided Status: New ** Also affects: python-oslo.privsep (Ubuntu Hirsute) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: python-oslo.privsep (Ubuntu Groovy) Importance: Undecided Status: New ** Changed in: neutron (Ubuntu Focal) Importance: Undecided => Medium ** Changed in: neutron (Ubuntu Focal) Status: New => Triaged ** Changed in: neutron (Ubuntu Groovy) Importance: Undecided => Medium ** Changed in: neutron (Ubuntu Groovy) Status: New => Triaged ** Changed in: neutron (Ubuntu Hirsute) Importance: Undecided => Medium ** Changed in: neutron (Ubuntu Hirsute) Status: New => Triaged ** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/ussuri Importance: Undecided Status: New ** Also affects: cloud-archive/victoria Importance: Undecided Status: New ** Changed in: cloud-archive/ussuri Importance: Undecided => Medium ** Changed in: cloud-archive/ussuri Status: New => Triaged ** Changed in: cloud-archive/victoria Importance: Undecided => Medium ** Changed in: cloud-archive/victoria Status: New => Triaged -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1896734 Title: A privsep daemon spawned by neutron-openvswitch-agent hangs when debug logging is enabled (large number of registered NICs) - an RPC response is too large for msgpack Status in OpenStack neutron-openvswitch charm: Invalid Status in Ubuntu Cloud Archive: Triaged Status in Ubuntu Cloud Archive ussuri series: Triaged Status in Ubuntu Cloud Archive victoria series: Triaged Status in neutron: Fix Released Status in oslo.privsep: New Status in neutron package in Ubuntu: Triaged Status in python-oslo.privsep package in Ubuntu: New Status in neutron source package in Focal: Triaged Status in python-oslo.privsep source package in Focal: New Status in neutron source package in Groovy: Triaged Status in python-oslo.privsep source package in Groovy: New Status in neutron source package in Hirsute: Triaged Status in python-oslo.privsep source package in Hirsute: New Bug description: When there is a large amount of netdevs registered in the kernel and debug logging is enabled, neutron-openvswitch-agent and the privsep daemon spawned by it hang since the RPC call result sent by the privsep daemon over a unix socket exceeds the message sizes that the msgpack library can handle. The impact of this is that enabling debug logging on the cloud completely stalls neutron-openvswitch-agents and makes them "dead" from the Neutron server perspective. The issue is summarized in detail in comment #5 https://bugs.launchpad.net/oslo.privsep/+bug/1896734/comments/5 Old Description While trying to debug a different issue, I encountered a situation where privsep hangs in the process of handling a request from neutron- openvswitch-agent when debug logging is enabled (juju debug-log neutron-openvswitch=true): https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1895652/comments/11 https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1895652/comments/12 The issue gets reproduced reliably in the environment where I encountered it on all units. As a result, neutron-openvswitch-agent services hang while waiting for a response from the privsep daemon and do not progress past basic initialization. They never post any state back to the Neutron server and thus are marked dead by it. The processes though are shown as "active (running)" by systemd which adds to the confusion since they do indeed start from the systemd's perspective. systemctl --no-pager status neutron-openvswitch-agent.service ● neutron-openvswitch-agent.service - Openstack Neutron Open vSwitch Plugin Agent Loaded: loaded (/lib/systemd/system/neutron-openvswitch-agent.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-09-23 08:28:41 UTC; 25min ago Main PID: 247772 (/usr/bin/python) Tasks: 4 (limit: 9830) CGroup: /system.slice/neutron-openvswitch-agent.service ├─247772 /usr/bin/python3 /usr/bin/neutron-openvswitch-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/openvswitch_…og └─248272 /usr/bin/python3 /usr/bin/privsep-helper --config-file
[Yahoo-eng-team] [Bug 1853613] Re: VMs don't get ip from dhcp after compute restart
** Also affects: neutron (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1853613 Title: VMs don't get ip from dhcp after compute restart Status in neutron: Fix Released Status in neutron package in Ubuntu: Incomplete Bug description: Env: pike + ovs + vxlan + l2pop + iptables_hybrid. Dhcp agent on differnt node than compute. Steps: 1. Boot 4 or more vms to same compute and same vxlan net. 2. Wait until they are fully running and reboot compute node. 3. After boot the vms are in status SHUTOFF. Start the vms. Vms don't get an ip address from neutron dhcp. The flood to tunnels flow (br-tun table 22) for the network is missing, so broadcasts like dhcp requests don't get on a tunnel to the node with dhcp agent. Neutron server did not send the flooding entry to the agent. It only does that for the first or second active port, or if the agent is restarted. After the compute boots, neutron-ovs-cleanup runs first and deletes the qvo ports from br-int [4]. Then the ovs-agent starts and nova- compute after it. Nova-compute destroys the domains and moves the vms to SHUTOFF status. It also (for some reason) recreates the qbr linux bridges and qvb/qvo veths connected to br-int. So neutron continues to see the ports as ACTIVE even though the vms are SHUTOFF, and agent_active_ports [1] never drops below 3. Also nova-compute might start a short time after the ovs-agent and the new ports are not detected in first iteration of the ovs agent loop, so agent_restarted will be false here [2]. Before [3] agent_restarted was true if the agent was running for less than agent_boot_time (default 180 sec) and the problem did not show. It does not happen if neutron-ovs-cleanup is disabled. Then the ovs agent first treats them as skipped_devices and they get status DOWN. [1] https://github.com/openstack/neutron/blob/21a52f7ae597f7992f32ff41cedff0c31e35c762/neutron/plugins/ml2/drivers/l2pop/mech_driver.py#L306 [2] https://github.com/openstack/neutron/blob/21a52f7ae597f7992f32ff41cedff0c31e35c762/neutron/plugins/ml2/drivers/l2pop/mech_driver.py#L310 [3] https://opendev.org/openstack/neutron/commit/62fe7852bbd70a24174853997096c52ee015e269 [4] https://bugs.launchpad.net/neutron/+bug/1853582 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1853613/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1864822] Re: Openvswitch Agent - Connexion openvswitch DB Broken
** Also affects: neutron (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1864822 Title: Openvswitch Agent - Connexion openvswitch DB Broken Status in neutron: Fix Released Status in neutron package in Ubuntu: Incomplete Bug description: Hi all, We have deployed more OpenStack plateform in my company. We used kolla ansible to deploy our plateforms. Here is the configuration that we applied : kolla_base_distro: "centos" kolla_install_type : "binary" openstack_version : "stein" Neutron architecture : HA l3 enable DVR enable SNAT Enabled multiple vlan provider : True Note: Our plateforms are multi-region Recently, we have upgraded a master region from rocky to stein with kolla ansible upgrade procedure. Since ugrade, sometimes openvswitch agent lost connexion to ovsdb. We have found this error in neutron-openvswitch-agent.log : "tcp:127.0.0.1:6640: send error: Broken pipe". And we have found this errors in ovsdb-server.log : 2020-02-24T23:13:22.644Z|9|reconnect|ERR|tcp:127.0.0.1:50260: no response to inactivity probe after 5 seconds, disconnecting 2020-02-25T04:10:55.893Z|00010|reconnect|ERR|tcp:127.0.0.1:58544: no response to inactivity probe after 5 seconds, disconnecting 2020-02-25T07:21:12.301Z|00011|reconnect|ERR|tcp:127.0.0.1:34918: no response to inactivity probe after 5 seconds, disconnecting 2020-02-25T09:21:45.533Z|00012|reconnect|ERR|tcp:127.0.0.1:37782: no response to inactivity probe after 5 seconds, disconnecting When we experience this issue, all "NORMAL" type flows inside br-ex doesn't get out. Example of flows stuck: (neutron-openvswitch-agent)[root@cnp69s12p07 /]# ovs-ofctl dump-flows br-ex | grep NORMAL cookie=0x7adbd675f988912b, duration=72705.077s, table=0, n_packets=185, n_bytes=16024, idle_age=65534, hard_age=65534, priority=0 actions=NORMAL cookie=0x7adbd675f988912b, duration=72695.007s, table=2, n_packets=11835702, n_bytes=5166123797, idle_age=0, hard_age=65534, priority=4,in_port=5,dl_vlan=1 actions=mod_vlan_vid:12,NORMAL cookie=0x7adbd675f988912b, duration=72694.928s, table=2, n_packets=4133243, n_bytes=349654412, idle_age=0, hard_age=65534, priority=4,in_port=5,dl_vlan=9 actions=mod_vlan_vid:18,NORMAL Workaround to solve this issue: - stop openvswitch_db openvswitch_vswitchd neutron_openvswitch_agent neutron_l3_agent (containers) - start containers: openvswitch_db openvswitch_vswitchd - start neutron_l3_agent neutron_openvswitch_agent Note: we have keep ovs connection timeout options by default : - of_connect_timeout: 300 - of_request_timeout: 300 - of_inactivity_probe: 10 Thank you in advance for your help. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1864822/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1829410] Re: vendor_data2 not supported
*** This bug is a duplicate of bug 1841104 *** https://bugs.launchpad.net/bugs/1841104 Marked as a duplicate: the good news is that we've released a fix for this! ** This bug has been marked a duplicate of bug 1841104 Openstack's vendor_data2.json is not handled -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1829410 Title: vendor_data2 not supported Status in cloud-init: Triaged Bug description: For the OpenStack datasource, I see some references to vendor_data2.json but it is not accessible via cloud.datasource.vendordata_pure or cloud.datasource.vendordata_raw or any other variable. Can full support be added for vendor_data2? It seems like it is supported, but looks like it is never requested or parsed. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1829410/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1829411] Re: vendor_data2 not supported
*** This bug is a duplicate of bug 1841104 *** https://bugs.launchpad.net/bugs/1841104 ** This bug is no longer a duplicate of bug 1829410 vendor_data2 not supported ** This bug has been marked a duplicate of bug 1841104 Openstack's vendor_data2.json is not handled -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1829411 Title: vendor_data2 not supported Status in cloud-init: New Bug description: For the OpenStack datasource, I see some references to vendor_data2.json but it is not accessible via cloud.datasource.vendordata_pure or cloud.datasource.vendordata_raw or any other variable. Can full support be added for vendor_data2? It seems like it is supported, but looks like it is never requested or parsed. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1829411/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1545702] Re: [OSSN-0088] Images v2 api metadef vulnerability
** Summary changed: - Images v2 api metadef vulnerability + [OSSN-0088] Images v2 api metadef vulnerability ** Also affects: ossn Importance: Undecided Status: New ** Changed in: ossn Status: New => Fix Released ** Changed in: ossn Importance: Undecided => Critical ** Changed in: ossn Assignee: (unassigned) => Abhishek Kekane (abhishek-kekane) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1545702 Title: [OSSN-0088] Images v2 api metadef vulnerability Status in Glance: Confirmed Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: Fix Released Bug description: It looks like a regular user can use the metadef api to create an unlimited number of records in the database. $ glance md-namespace-create ns1 xxx $ glance md-namespace-create ns2 xxx . . . $ glance md-tag-create --name tag OS::Software::WebServers $ glance md-tag-create --name tag2 OS::Software::WebServers . . . etc. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1545702/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1916922] Re: Glance leaks resource types across namespaces
We'll be switching this bug public shortly along with bug 1916926 under a single publication (OSSN-0088). ** Description changed: - This issue is being treated as a potential security risk under - embargo. Please do not make any public mention of embargoed - (private) security vulnerabilities before their coordinated - publication by the OpenStack Vulnerability Management Team in the - form of an official OpenStack Security Advisory. This includes - discussion of the bug or associated fixes in public forums such as - mailing lists, code review systems and bug trackers. Please also - avoid private disclosure to other individuals not already approved - for access to this information, and provide this same reminder to - those who are made aware of the issue prior to publication. All - discussion should remain confined to this private bug report, and - any proposed fixes should be added to the bug as attachments. This - embargo shall not extend past 2021-05-26 and will be made - public by or on that date even if no fix is identified. - - - As a user of a project, I can see resource types associated to private namespaces I don't have access to: + As a user of a project, I can see resource types associated to private + namespaces I don't have access to: ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ cat alicerc export OS_CACERT= export OS_PROJECT_NAME=separate export OS_USERNAME=alice export OS_PASSWORD=password export OS_REGION_NAME=RegionOne export OS_IDENTITY_API_VERSION=3 export OS_AUTH_TYPE=password export OS_AUTH_URL=http://192.168.1.155/identity export OS_USER_DOMAIN_ID=default export OS_PROJECT_DOMAIN_ID=default export OS_VOLUME_API_VERSION=3 ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ source alicerc ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ glance md-namespace-list +--+ | namespace| +--+ | OS::Software::DBMS | | CIM::ResourceAllocationSettingData | | OS::Compute::CPUPinning | | OS::Compute::Watchdog| | OS::Compute::GuestMemoryBacking | | OS::Compute::AggregateDiskFilter | | OS::Compute::RandomNumberGenerator | | OS::Compute::Hypervisor | | OS::Compute::AggregateIoOpsFilter| | OS::Compute::VirtCPUTopology | | OS::Compute::HostCapabilities| | CIM::ProcessorAllocationSettingData | | OS::Compute::GuestShutdownBehavior | | OS::Cinder::Volumetype | | OS::Software::WebServers | | OS::Compute::Libvirt | | OS::Compute::XenAPI | | OS::Compute::Quota | | OS::Compute::VMwareFlavor| | OS::Compute::VMwareQuotaFlavor | | OS::Compute::InstanceData| | OS::Compute::LibvirtImage| | OS::Compute::AggregateNumInstancesFilter | | OS::Glance::Signatures | | CIM::VirtualSystemSettingData| | CIM::StorageAllocationSettingData| | OS::Software::Runtimes | | OS::Compute::VMware | +--+ ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ glance md-namespace-create alice-namespace ++--+ | Property | Value| ++--+ | created_at | 2021-02-25T15:55:55Z | | namespace | alice-namespace | | owner | 67f1495e5dc145abbfa7059c63c6eda2 | | protected | False| | schema | /v2/schemas/metadefs/namespace | | updated_at | 2021-02-25T15:55:55Z | | visibility | private | ++--+ ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ glance md-resource-type-associate --name alice-resource-type alice-namespace ++--+ | Property | Value| ++--+ | created_at | 2021-02-25T15:57:29Z | | name | alice-resource-type | | updated_at | 2021-02-25T15:57:29Z | ++--+ Now as a separate user ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ source openrc demo demo is_service_enabled:29: command not found: set +o xtrace WARNING: setting legacy OS_TENANT_NAME to support cli tools. ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ glance md-resource-type-list +-+ | name| +-+ | OS::Glance::Image | | OS::Cinder::Volume | | OS::Nova::Server| | OS::Nova::Aggregate | | OS::Nova::Flavor| |
[Yahoo-eng-team] [Bug 1901527] Re: Agent API gets broken if OVN DB is upgraded to the one introducing chassis_private
Yes, I think it's https://review.opendev.org/c/openstack/neutron/+/760967 but you're right, it's fixed now. Thanks for pointing it out! ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1901527 Title: Agent API gets broken if OVN DB is upgraded to the one introducing chassis_private Status in neutron: Fix Released Bug description: Neutron determines which table to use for agent healthchecks at the initialization phase. With OVN databases that don't implement chassis_private table a chassis table is used, which has scaling impacts. However, if OVN DB is upgraded while Neutron is running, the healthchecks will be still performed on chassis table while OVN already uses chassis_private. This leads to reporting dead agents by the API. We should detect OVN was upgraded and start using chassis_private table since. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1901527/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1916926] Re: Glance leaks namespace existence to unauthorized users
** Description changed: - This issue is being treated as a potential security risk under - embargo. Please do not make any public mention of embargoed - (private) security vulnerabilities before their coordinated - publication by the OpenStack Vulnerability Management Team in the - form of an official OpenStack Security Advisory. This includes - discussion of the bug or associated fixes in public forums such as - mailing lists, code review systems and bug trackers. Please also - avoid private disclosure to other individuals not already approved - for access to this information, and provide this same reminder to - those who are made aware of the issue prior to publication. All - discussion should remain confined to this private bug report, and - any proposed fixes should be added to the bug as attachments. This - embargo shall not extend past 2021-05-26 and will be made - public by or on that date even if no fix is identified. - ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ source openrc demo demo ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ openstack token issue ++-+ | Field | Value | ++-+ | expires| 2021-02-25T14:11:38+ | | id | gABgN6IKDUKTn9RNudtZD605vA9l9eErCcXDrdxZwfhePYVlAHXzzCdQs6FK6XDwFvuexzfymc0uX7NY5RisEnQmUBl6eLccgBMYE6vSpVWCDTkFuKIuPfLh3xSkJGjZcpG7nfJ_ImU_wCJJFgcclf1zHTHWQ9Y15k-mAE7l3xceqUkOx2Y | | project_id | ed4fade2e2cd4be0932ef30357f6d7a1 | | user_id| e83b2f50463c4959bcc00a96b52b2f86 | ++-+ ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ glance md-namespace-show foo ++--+ | Property | Value| ++--+ | created_at | 2021-02-25T04:54:10Z | | namespace | foo | | owner | ed4fade2e2cd4be0932ef30357f6d7a1 | | protected | False| | resource_type_associations | ["bar"] | | schema | /v2/schemas/metadefs/namespace | | updated_at | 2021-02-25T04:54:10Z | | visibility | private | ++--+ ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ source alicerc ╭─ubuntu@glance-devstack ~/devstack ‹master*› ╰─➤ $ glance md-resource-type-associate --name test foo HTTP 403 Forbidden: Forbidding request, metadata definition namespace=foo is not visible. This might not be a security issue since the user needs to know the namespace name, but opening this in private based on a recommendation from jokke. ** Information type changed from Private Security to Public ** Summary changed: - Glance leaks namespace existence to unauthorized users + [OSSN-0088] Glance leaks namespace existence to unauthorized users ** Changed in: ossa Status: Incomplete => Won't Fix ** Also affects: ossn Importance: Undecided Status: New ** Changed in: ossn Status: New => Fix Released ** Changed in: ossn Importance: Undecided => Critical ** Changed in: ossn Assignee: (unassigned) => Abhishek Kekane (abhishek-kekane) ** Tags added: security -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1916926 Title: [OSSN-0088] Glance leaks namespace existence to unauthorized users Status in Glance: New Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: Fix
[Yahoo-eng-team] [Bug 1896203] Re: TypeError: Cannot look up record by empty string in check_for_igmp_snoop_support task
Fixed by https://review.opendev.org/c/openstack/neutron/+/758391
** Changed in: neutron
Status: In Progress => Fix Released
** Changed in: neutron
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1896203
Title:
TypeError: Cannot look up record by empty string in
check_for_igmp_snoop_support task
Status in neutron:
Fix Released
Bug description:
openstack ussuri deployment
neutron: 16.1.0
ovn: 20.06.2
ovs: 2.13.1
We are seeing the following traceback for one of maintenance tasks:
--8<--8<--8<--
2020-09-18 08:49:11.572 88 ERROR futurist.periodics
[req-036b9eae-53cf-4c55-a946-8aa2dc6697c5 - - - - -] Failed to call periodic
'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support'
(it runs every 600.00 seconds): TypeError: Cannot look up record by empty
string
2020-09-18 08:49:11.572 88 ERROR futurist.periodics Traceback (most recent
call last):
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/futurist/periodics.py", line
293, in run
2020-09-18 08:49:11.572 88 ERROR futurist.periodics work()
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/futurist/periodics.py", line
67, in __call__
2020-09-18 08:49:11.572 88 ERROR futurist.periodics return
self.callback(*self.args, **self.kwargs)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/futurist/periodics.py", line
181, in decorator
2020-09-18 08:49:11.572 88 ERROR futurist.periodics return f(*args,
**kwargs)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py",
line 551, in check_for_igmp_snoop_support
2020-09-18 08:49:11.572 88 ERROR futurist.periodics
ovn_const.MCAST_FLOOD_UNREGISTERED: value})))
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2020-09-18 08:49:11.572 88 ERROR futurist.periodics next(self.gen)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py",
line 185, in transaction
2020-09-18 08:49:11.572 88 ERROR futurist.periodics yield t
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2020-09-18 08:49:11.572 88 ERROR futurist.periodics next(self.gen)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in
transaction
2020-09-18 08:49:11.572 88 ERROR futurist.periodics del
self._nested_txns_map[cur_thread_id]
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in
__exit__
2020-09-18 08:49:11.572 88 ERROR futurist.periodics self.result =
self.commit()
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py",
line 62, in commit
2020-09-18 08:49:11.572 88 ERROR futurist.periodics raise result.ex
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py",
line 122, in run
2020-09-18 08:49:11.572 88 ERROR futurist.periodics
txn.results.put(txn.do_commit())
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py",
line 89, in do_commit
2020-09-18 08:49:11.572 88 ERROR futurist.periodics command.run_idl(txn)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py",
line 134, in run_idl
2020-09-18 08:49:11.572 88 ERROR futurist.periodics record =
self.api.lookup(self.table, self.record)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py",
line 107, in lookup
2020-09-18 08:49:11.572 88 ERROR futurist.periodics return
self._lookup(table, record)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py",
line 115, in _lookup
2020-09-18 08:49:11.572 88 ERROR futurist.periodics raise
TypeError("Cannot look up record by empty string")
2020-09-18 08:49:11.572 88 ERROR futurist.periodics TypeError: Cannot look up
record by empty
[Yahoo-eng-team] [Bug 1918292] [NEW] rescuing a boot from volume instance failed with "no bootable device" when hw_rescue_bus=scsi
Public bug reported: I spent some time understanding how to rescue instances which are boot- from-volume. After finding a setup which would rescue instances properly with hw_rescue_device=disk and hw_rescue_bus=usb, switching the rescue bus to hw_rescue_bus=scsi resulted in an Ubuntu Focal instance to show "No bootable device" in the instance console when trying to rescue it. Switching the bus back to USB and unrescue/rescue worked as expected. Here is the xml from the scsi bus rescue attempt http://paste.openstack.org/show/803383/ Commands like this were used: openstack server rescue --image d0d00858-0672-4d19-9e5a-fb120d78c0fc 81057513-24cd- 49e7-b1e5-939d4080bb21 --os-compute-api-version 2.87 Environment === 1. Exact version of OpenStack you are running. OpenStack Victoria, OpenStack-Ansible 22.0.0 2. Which hypervisor did you use? Libvirt + KVM. libvirt 6.0.0-0ubuntu8.5~cloud0 2. Which storage type did you use? All storage on Ceph 3. Which networking type did you use? Neutron with LinuxBridge ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1918292 Title: rescuing a boot from volume instance failed with "no bootable device" when hw_rescue_bus=scsi Status in OpenStack Compute (nova): New Bug description: I spent some time understanding how to rescue instances which are boot-from-volume. After finding a setup which would rescue instances properly with hw_rescue_device=disk and hw_rescue_bus=usb, switching the rescue bus to hw_rescue_bus=scsi resulted in an Ubuntu Focal instance to show "No bootable device" in the instance console when trying to rescue it. Switching the bus back to USB and unrescue/rescue worked as expected. Here is the xml from the scsi bus rescue attempt http://paste.openstack.org/show/803383/ Commands like this were used: openstack server rescue --image d0d00858-0672-4d19-9e5a-fb120d78c0fc 81057513-24cd- 49e7-b1e5-939d4080bb21 --os-compute-api-version 2.87 Environment === 1. Exact version of OpenStack you are running. OpenStack Victoria, OpenStack-Ansible 22.0.0 2. Which hypervisor did you use? Libvirt + KVM. libvirt 6.0.0-0ubuntu8.5~cloud0 2. Which storage type did you use? All storage on Ceph 3. Which networking type did you use? Neutron with LinuxBridge To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1918292/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918288] [NEW] rescue of boot from volume image fails when --image is not specified
Public bug reported:
>From an environment where all storage is on ceph, it was not possible to
omit the --image parameter when rescuing a boot-from-volume instance as
suggested near the end of this doc:
https://docs.openstack.org/nova/latest/user/rescue.html
The doc implies that omitting --image will use the image of the rescued
instance as the rescue image.
Works:
openstack server rescue --image d0d00858-0672-4d19-9e5a-fb120d78c0fc
81057513-24cd-49e7-b1e5-939d4080bb21 --os-compute-api-version 2.87
Does not work:
openstack server rescue 81057513-24cd-49e7-b1e5-939d4080bb21 --os-
compute-api-version 2.87
Not specifying --image appears to succeed with the command line client:
REQ: curl -g -i --insecure -X POST
http://10.11.128.30:8774/v2.1/servers/81057513-24cd-49e7-b1e5-939d4080bb21/action
-H "Accept: application/json" -H "Content-Type: application/json" -H
"OpenStack-API-Version: compute 2.87" -H "User-Agent: python-novaclient" -H
"X-Auth-Token:
{SHA256}9418545bce89f055d82cf493560aaa6c1e0fb0180f7bae5615006283e83dd999" -H
"X-OpenStack-Nova-API-Version: 2.87" -d '{"rescue": null}'
http://10.11.128.30:8774 "POST
/v2.1/servers/81057513-24cd-49e7-b1e5-939d4080bb21/action HTTP/1.1" 200 29
However in the nova compute log we see this stacktrace
http://paste.openstack.org/show/803379/
Environment
===
1. OpenStack Victoria / OpenStack-Ansible 22.0.0
2. Which hypervisor did you use?
Libvirt + KVM libvirt 6.0.0-0ubuntu8.5~cloud0
2. Which storage type did you use? Ceph
3. Which networking type did you use?
Neutron / LinuxBridge
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1918288
Title:
rescue of boot from volume image fails when --image is not specified
Status in OpenStack Compute (nova):
New
Bug description:
From an environment where all storage is on ceph, it was not possible
to omit the --image parameter when rescuing a boot-from-volume
instance as suggested near the end of this doc:
https://docs.openstack.org/nova/latest/user/rescue.html
The doc implies that omitting --image will use the image of the
rescued instance as the rescue image.
Works:
openstack server rescue --image d0d00858-0672-4d19-9e5a-fb120d78c0fc
81057513-24cd-49e7-b1e5-939d4080bb21 --os-compute-api-version 2.87
Does not work:
openstack server rescue 81057513-24cd-49e7-b1e5-939d4080bb21 --os-
compute-api-version 2.87
Not specifying --image appears to succeed with the command line
client:
REQ: curl -g -i --insecure -X POST
http://10.11.128.30:8774/v2.1/servers/81057513-24cd-49e7-b1e5-939d4080bb21/action
-H "Accept: application/json" -H "Content-Type: application/json" -H
"OpenStack-API-Version: compute 2.87" -H "User-Agent: python-novaclient" -H
"X-Auth-Token:
{SHA256}9418545bce89f055d82cf493560aaa6c1e0fb0180f7bae5615006283e83dd999" -H
"X-OpenStack-Nova-API-Version: 2.87" -d '{"rescue": null}'
http://10.11.128.30:8774 "POST
/v2.1/servers/81057513-24cd-49e7-b1e5-939d4080bb21/action HTTP/1.1" 200 29
However in the nova compute log we see this stacktrace
http://paste.openstack.org/show/803379/
Environment
===
1. OpenStack Victoria / OpenStack-Ansible 22.0.0
2. Which hypervisor did you use?
Libvirt + KVM libvirt 6.0.0-0ubuntu8.5~cloud0
2. Which storage type did you use? Ceph
3. Which networking type did you use?
Neutron / LinuxBridge
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1918288/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918274] [NEW] Driver VLAN do not create the VlanAllocation registers if "network_vlan_ranges" only specify the network
Public bug reported:
"network_vlan_ranges" allows to define a list of physical networks and
VLAN ranges [1]. But is possible to just define the physical network
without the VLAN ID ranges; in that case, the VLAN driver should take
all valid VLAN IDs (from 1 to 4094).
If no VLAN range is defined, when "VlanTypeDriver._sync_vlan_allocations" is
called, the "ranges" variable is:
ranges = {'physical_net': []}
We have two options here:
- Modify the sync method.
- Change the parser to populate, in case of not passing any VLAN range, the
whole list of possible VLAN IDs for this physnet (I think last option is
better).
[1]https://github.com/openstack/neutron/blob/ab78d29c68b3cfdc6732d2ed2bcec0677b1a34f3/neutron/conf/plugins/ml2/drivers/driver_type.py#L68-L76
** Affects: neutron
Importance: Undecided
Assignee: Rodolfo Alonso (rodolfo-alonso-hernandez)
Status: New
** Changed in: neutron
Assignee: (unassigned) => Rodolfo Alonso (rodolfo-alonso-hernandez)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1918274
Title:
Driver VLAN do not create the VlanAllocation registers if
"network_vlan_ranges" only specify the network
Status in neutron:
New
Bug description:
"network_vlan_ranges" allows to define a list of physical networks and
VLAN ranges [1]. But is possible to just define the physical network
without the VLAN ID ranges; in that case, the VLAN driver should take
all valid VLAN IDs (from 1 to 4094).
If no VLAN range is defined, when "VlanTypeDriver._sync_vlan_allocations" is
called, the "ranges" variable is:
ranges = {'physical_net': []}
We have two options here:
- Modify the sync method.
- Change the parser to populate, in case of not passing any VLAN range, the
whole list of possible VLAN IDs for this physnet (I think last option is
better).
[1]https://github.com/openstack/neutron/blob/ab78d29c68b3cfdc6732d2ed2bcec0677b1a34f3/neutron/conf/plugins/ml2/drivers/driver_type.py#L68-L76
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1918274/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1886956] Re: Functional test test_restart_wsgi_on_sighup_multiple_workers is failing sometimes
After https://review.opendev.org/c/openstack/neutron/+/778080 was merged
I didn't saw it anymore in CI. So I'm closing that LP now.
** Changed in: neutron
Status: Incomplete => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1886956
Title:
Functional test test_restart_wsgi_on_sighup_multiple_workers is
failing sometimes
Status in neutron:
Fix Released
Bug description:
Test
neutron.tests.functional.test_server.TestWsgiServer.test_restart_wsgi_on_sighup_multiple_workers
is failing from time to time.
Logstash query:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20_test_restart_service_on_sighup%5C%22
Failure example:
https://f7a63aeb9edd557a2176-4740624f0848c8c3257f704064a4516f.ssl.cf2.rackcdn.com/736026/4/gate
/neutron-functional/d7d5c47/testr_results.html
ft1.1:
neutron.tests.functional.test_server.TestWsgiServer.test_restart_wsgi_on_sighup_multiple_workerstesttools.testresult.real._StringException:
Traceback (most recent call last):
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line
704, in wait_until_true
eventlet.sleep(sleep)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/greenthread.py",
line 36, in sleep
hub.switch()
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py",
line 298, in switch
return self.greenlet.switch()
eventlet.timeout.Timeout: 5 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py",
line 182, in func
return f(self, *args, **kwargs)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/test_server.py",
line 237, in test_restart_wsgi_on_sighup_multiple_workers
workers=2)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/test_server.py",
line 163, in _test_restart_service_on_sighup
utils.wait_until_true(condition, timeout=5, sleep=1)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line
709, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 5 seconds
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1886956/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1881095] Re: [OVN] Router availability zones support
** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1881095 Title: [OVN] Router availability zones support Status in neutron: Fix Released Bug description: Reference: https://docs.openstack.org/neutron/latest/admin/config- az.html For feature parity, OVN needs to add support for the "router_availability_zone" extension. This means that, when scheduling the logical router ports the OVN driver should take in consideration the router availability zones hints and only schedule the ports onto the nodes (or chassis in OVN terms) that belongs to those availability zones. Differently from ML2/OVS, the OVN driver does not have L3 agents running on nodes such as networker nodes so we can't reuse the same agents configuration option. In OVN we will need to find another place to put this information. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1881095/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1898862] Re: Job neutron-ovn-tempest-ovs-release-ipv6-only is failing 100% of times
Job has been fixed at DevStack: https://review.opendev.org/c/openstack/devstack/+/753575 ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1898862 Title: Job neutron-ovn-tempest-ovs-release-ipv6-only is failing 100% of times Status in neutron: Fix Released Bug description: Every time I see 9 tests failing. Like e.g. https://zuul.opendev.org/t/openstack/build/8a783bb50f724c6f8d906c939c2bdb1e To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1898862/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1909038] Re: [ovn] TypeError: lrp_set_options() takes 2 positional arguments but 3 were given
Fixed at: https://review.opendev.org/c/openstack/neutron/+/768277 ** Changed in: neutron Status: New => Fix Committed ** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1909038 Title: [ovn] TypeError: lrp_set_options() takes 2 positional arguments but 3 were given Status in neutron: Fix Released Bug description: I see the following error when restarting the neutron process on master devstack deployment: TypeError: lrp_set_options() takes 2 positional arguments but 3 were given Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics [None req-be925f41-7b5f-4e04-893a-282cdbb5d137 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_fragmentation_support' (it runs every 600.00 seconds): TypeError: lrp_set_options() takes 2 positional arguments but 3 were given Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics Traceback (most recent call last): Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics File "/usr/local/lib/python3.8/dist-packages/futurist/periodics.py", line 293, in run Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics work() Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics File "/usr/local/lib/python3.8/dist-packages/futurist/periodics.py", line 67, in __call__ Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics File "/usr/local/lib/python3.8/dist-packages/futurist/periodics.py", line 181, in decorator Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics return f(*args, **kwargs) Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 548, in check_for_fragmentation_support Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics self._ovn_client.set_gateway_mtu(context, net) Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1624, in set_gateway_mtu Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics commands.append(self._nb_idl.lrp_set_options(lrp_name, options)) Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics TypeError: lrp_set_options() takes 2 positional arguments but 3 were given Dec 22 21:02:06 stack neutron-server[200078]: ERROR futurist.periodics To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1909038/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1918266] [NEW] Functional test test_gateway_chassis_rebalance failing due to "failed to bind logical router"
Public bug reported: Error example: https://40b5766ce602bfb4b663-445d4465f34d2b24df5d805a76ff9803.ssl.cf1.rackcdn.com/765846/7/check /neutron-functional-with-uwsgi/928e7cf/testr_results.html Stacktrace: ft1.14: neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalancetesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/ovn_l3/test_plugin.py", line 497, in test_gateway_chassis_rebalance self.assertTrue(self.cr_lrp_pb_event.wait(logical_port), File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/unittest2/case.py", line 702, in assertTrue raise self.failureException(msg) AssertionError: False is not true : lrp cr-lrp-488b3887-1770-4a72-86cb-7306e78c954a failed to bind Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%20497%2C%20in%20test_gateway_chassis_rebalance%5C%22 ** Affects: neutron Importance: High Status: Confirmed ** Tags: functional-tests gate-failure ovn -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1918266 Title: Functional test test_gateway_chassis_rebalance failing due to "failed to bind logical router" Status in neutron: Confirmed Bug description: Error example: https://40b5766ce602bfb4b663-445d4465f34d2b24df5d805a76ff9803.ssl.cf1.rackcdn.com/765846/7/check /neutron-functional-with-uwsgi/928e7cf/testr_results.html Stacktrace: ft1.14: neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalancetesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/ovn_l3/test_plugin.py", line 497, in test_gateway_chassis_rebalance self.assertTrue(self.cr_lrp_pb_event.wait(logical_port), File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/unittest2/case.py", line 702, in assertTrue raise self.failureException(msg) AssertionError: False is not true : lrp cr-lrp-488b3887-1770-4a72-86cb-7306e78c954a failed to bind Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%20497%2C%20in%20test_gateway_chassis_rebalance%5C%22 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1918266/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1896734] Re: A privsep daemon spawned by neutron-openvswitch-agent hangs when debug logging is enabled (large number of registered NICs) - an RPC response is too large for msgpac
Neutron fix is released upstream and backported all the way to Train. ** Changed in: neutron Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1896734 Title: A privsep daemon spawned by neutron-openvswitch-agent hangs when debug logging is enabled (large number of registered NICs) - an RPC response is too large for msgpack Status in OpenStack neutron-openvswitch charm: Invalid Status in neutron: Fix Released Status in oslo.privsep: New Status in python-oslo.privsep package in Ubuntu: New Bug description: When there is a large amount of netdevs registered in the kernel and debug logging is enabled, neutron-openvswitch-agent and the privsep daemon spawned by it hang since the RPC call result sent by the privsep daemon over a unix socket exceeds the message sizes that the msgpack library can handle. The impact of this is that enabling debug logging on the cloud completely stalls neutron-openvswitch-agents and makes them "dead" from the Neutron server perspective. The issue is summarized in detail in comment #5 https://bugs.launchpad.net/oslo.privsep/+bug/1896734/comments/5 Old Description While trying to debug a different issue, I encountered a situation where privsep hangs in the process of handling a request from neutron- openvswitch-agent when debug logging is enabled (juju debug-log neutron-openvswitch=true): https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1895652/comments/11 https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1895652/comments/12 The issue gets reproduced reliably in the environment where I encountered it on all units. As a result, neutron-openvswitch-agent services hang while waiting for a response from the privsep daemon and do not progress past basic initialization. They never post any state back to the Neutron server and thus are marked dead by it. The processes though are shown as "active (running)" by systemd which adds to the confusion since they do indeed start from the systemd's perspective. systemctl --no-pager status neutron-openvswitch-agent.service ● neutron-openvswitch-agent.service - Openstack Neutron Open vSwitch Plugin Agent Loaded: loaded (/lib/systemd/system/neutron-openvswitch-agent.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-09-23 08:28:41 UTC; 25min ago Main PID: 247772 (/usr/bin/python) Tasks: 4 (limit: 9830) CGroup: /system.slice/neutron-openvswitch-agent.service ├─247772 /usr/bin/python3 /usr/bin/neutron-openvswitch-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/openvswitch_…og └─248272 /usr/bin/python3 /usr/bin/privsep-helper --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini -…ck An strace shows that the privsep daemon tries to receive input from fd 3 which is the unix socket it uses to communicate with the client. However, this is just one tread out of many spawned by the privsep daemon so it is unlikely to be the root cause (there are 65 threads there in total, see https://paste.ubuntu.com/p/fbGvN2P8rP/) # there is one extra neutron-openvvswitch-agent running in a LXD container which can be ignored here (there is an octavia unit on the node which has a neutron-openvswitch subordinate) root@node2:~# ps -eo pid,user,args --sort user | grep -P 'privsep.*openvswitch' 860690 10 /usr/bin/python3 /usr/bin/privsep-helper --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --privsep_context neutron.privileged.default --privsep_sock_path /tmp/tmp910qakfk/privsep.sock 248272 root /usr/bin/python3 /usr/bin/privsep-helper --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --privsep_context neutron.privileged.default --privsep_sock_path /tmp/tmpcmwn7vom/privsep.sock 363905 root grep --color=auto -P privsep.*openvswitch root@node2:~# strace -f -p 248453 2>&1 [pid 248786] futex(0x7f6a6401c1d0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0x [pid 248475] futex(0x7f6a6c024590, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0x [pid 248473] futex(0x7f6a746d9fd0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0x [pid 248453] recvfrom(3, root@node2:~# lsof -p 248453 | grep 3u privsep-h 248453 root3u unix 0x8e6d8abdec00 0t0 356522977 type=STREAM root@node2:~# ss -pax | grep 356522977 u_str ESTAB 00
[Yahoo-eng-team] [Bug 1918255] [NEW] Race between sriov_config service and openib service
Public bug reported: When having sriov configured in mlnx interfaces and we are using MLNX ofed, In reboot case we'll have race between sriov_config service and openibd service which loads mlx drivers, sometimes it causes failures in starting sriov_config service correctly ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1918255 Title: Race between sriov_config service and openib service Status in neutron: New Bug description: When having sriov configured in mlnx interfaces and we are using MLNX ofed, In reboot case we'll have race between sriov_config service and openibd service which loads mlx drivers, sometimes it causes failures in starting sriov_config service correctly To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1918255/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
