[Yahoo-eng-team] [Bug 1833385] Re: Functional tests are failing due to missing namespace
[Expired for neutron because there has been no activity for 60 days.] ** Changed in: neutron Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1833385 Title: Functional tests are failing due to missing namespace Status in neutron: Expired Bug description: Example of failure: http://logs.openstack.org/48/665548/4/check/neutron- functional/f6d6447/testr_results.html.gz But from journal log in this job's results it looks that this missing namespace was existing earlier. So maybe it's some race condition or slow down of node? It has to be checked. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1833385/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1858661] Re: Error when creating a Linux interface
[Expired for neutron because there has been no activity for 60 days.] ** Changed in: neutron Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1858661 Title: Error when creating a Linux interface Status in neutron: Expired Bug description: A recurrent error, from time to time, in the CI. When a Linux interface is being created, the test case raises a timeout exception. This is happening during the execution of https://github.com/openstack/neutron/blob/ac63c570a1c630ac4405e4caf3d516d069165d69/neutron/agent/linux/bridge_lib.py#L63. Logs: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ea8/677092/11/check/neutron-functional/ea8fe46/testr_results.html.gz Error: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/fixtures/fixture.py", line 197, in setUp self._setUp() File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 882, in _setUp self.bridge = self._create_bridge() File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 909, in _create_bridge namespace=self.namespace) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/base.py", line 47, in create_resource return creation_func(name, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/bridge_lib.py", line 63, in addbr bridge.link.create() File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 470, in create self.kind) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 245, in _wrap return self.channel.remote_call(name, args, kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 194, in remote_call result = self.send_recv((Message.CALL.value, name, args, kwargs)) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/oslo_privsep/comm.py", line 171, in send_recv reply = future.result() File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/oslo_privsep/comm.py", line 108, in result self.condvar.wait() File "/usr/lib/python3.6/threading.py", line 295, in wait waiter.acquire() File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/semaphore.py", line 115, in acquire hubs.get_hub().switch() File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch return self.greenlet.switch() File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 350, in run self.wait(sleep_time) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 80, in wait presult = self.do_poll(seconds) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll return self.poll.poll(seconds) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler raise TimeoutException() fixtures._fixtures.timeout.TimeoutException To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1858661/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1836642] Re: Metadata responses are very slow sometimes
[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.] ** Changed in: nova Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1836642 Title: Metadata responses are very slow sometimes Status in neutron: Incomplete Status in OpenStack Compute (nova): Expired Bug description: It happens from time to time in CI that VM spawned in test don't get public-keys from metadata service. Due to that SSH to instance using ssh-key is not possible thus test fails. Example of such failures: http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/testr_results.html.gz http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/testr_results.html.gz http://logs.openstack.org/45/645645/8/gate/tempest-full/4d7874e/testr_results.html.gz In each of those cases it looks that neutron metadata agent was waiting more than 10 seconds for response from n-api-meta service: http://logs.openstack.org/09/666409/7/check/tempest- full/08f4c53/controller/logs/screen-n-api- meta.txt.gz#_Jul_11_23_43_16_704979 ~ 16 seconds, http://logs.openstack.org/35/521035/8/check/tempest- full/031b0b9/controller/logs/screen-n-api- meta.txt.gz#_Jul_09_17_23_47_871054 ~ 13 seconds, http://logs.openstack.org/45/645645/8/gate/tempest- full/4d7874e/controller/logs/screen-n-api- meta.txt.gz#_Jul_09_01_43_56_549916 ~ 17 seconds. I have no idea why nova is responding so slow but it would be worth if someone from Nova team would take a look into that. Logstash query which can help to find another examples: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22failed%20to%20get%20http%3A%2F%2F169.254.169.254%2F2009-04-04 %2Fmeta-data%2Fpublic-keys%5C%22 It is possible that in some cases of failed tests, reason of failure may be different but problem described above is quite common in those failed tests IMO. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1836642/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1837552] Re: neutron-tempest-with-uwsgi job finish with timeout very often
[Expired for neutron because there has been no activity for 60 days.] ** Changed in: neutron Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1837552 Title: neutron-tempest-with-uwsgi job finish with timeout very often Status in neutron: Expired Bug description: Example of such timeouted job: http://logs.openstack.org/22/660722/15/check/neutron-tempest-with- uwsgi/721b94f/job-output.txt.gz I checked only this one example failure so far but it looks for me that all was running fine for some time and than suddently all tests started failing, exactly since: http://logs.openstack.org/22/660722/15/check/neutron-tempest-with- uwsgi/721b94f/job-output.txt.gz#_2019-07-22_22_43_40_285453 In apache logs I see that around this time many "500" responses started happening: http://logs.openstack.org/22/660722/15/check/neutron-tempest-with- uwsgi/721b94f/controller/logs/apache/access_log.txt.gz But in neutron-api logs there is nothing wrong: http://logs.openstack.org/22/660722/15/check/neutron-tempest-with-uwsgi/721b94f/controller/logs/screen-neutron-api.txt.gz It has to be investigated why it happens like that. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1837552/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1813959] Re: test_hotplug_nic intermittently fails teardown due to "One or more ports have an IP allocation from this subnet."
[Expired for neutron because there has been no activity for 60 days.]
** Changed in: neutron
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1813959
Title:
test_hotplug_nic intermittently fails teardown due to "One or more
ports have an IP allocation from this subnet."
Status in neutron:
Expired
Bug description:
Seen here:
http://logs.openstack.org/92/627892/9/check/nova-next/d402065/job-
output.txt.gz#_2019-01-30_03_05_59_790327
2019-01-30 03:05:59.790327 | primary | Captured traceback-1:
2019-01-30 03:05:59.790348 | primary | ~
2019-01-30 03:05:59.790375 | primary | Traceback (most recent call last):
2019-01-30 03:05:59.790440 | primary | File
"tempest/lib/common/utils/test_utils.py", line 84, in
call_and_ignore_notfound_exc
2019-01-30 03:05:59.790467 | primary | return func(*args, **kwargs)
2019-01-30 03:05:59.790518 | primary | File
"tempest/lib/services/network/subnets_client.py", line 52, in delete_subnet
2019-01-30 03:05:59.790549 | primary | return
self.delete_resource(uri)
2019-01-30 03:05:59.790607 | primary | File
"tempest/lib/services/network/base.py", line 41, in delete_resource
2019-01-30 03:05:59.790636 | primary | resp, body =
self.delete(req_uri)
2019-01-30 03:05:59.790688 | primary | File
"tempest/lib/common/rest_client.py", line 311, in delete
2019-01-30 03:05:59.790733 | primary | return self.request('DELETE',
url, extra_headers, headers, body)
2019-01-30 03:05:59.790774 | primary | File
"tempest/lib/common/rest_client.py", line 676, in request
2019-01-30 03:05:59.790805 | primary | self._error_checker(resp,
resp_body)
2019-01-30 03:05:59.790850 | primary | File
"tempest/lib/common/rest_client.py", line 797, in _error_checker
2019-01-30 03:05:59.790885 | primary | raise
exceptions.Conflict(resp_body, resp=resp)
2019-01-30 03:05:59.790930 | primary | tempest.lib.exceptions.Conflict:
Conflict with state of target resource
2019-01-30 03:05:59.791051 | primary | Details: {u'detail': u'', u'type':
u'SubnetInUse', u'message': u'Unable to complete operation on subnet
e6e7e892-bd6b-453b-9e6f-08d889c83190: One or more ports have an IP allocation
from this subnet.'}
From the server logs:
http://logs.openstack.org/92/627892/9/check/nova-
next/d402065/logs/screen-q-svc.txt.gz#_Jan_30_03_05_44_715787
Jan 30 03:05:44.715787 ubuntu-xenial-ovh-bhs1-0002242443 neutron-
server[25564]: INFO neutron.db.db_base_plugin_v2 [None req-
62a95088-0965-4d58-a43d-6450fee48399 tempest-
TestNetworkBasicOps-1823641274 tempest-TestNetworkBasicOps-1823641274]
Found port (3080b6d8-f622-496b-a749-0b7def78fb6c, 10.1.0.26) having IP
allocation on subnet e6e7e892-bd6b-453b-9e6f-08d889c83190, cannot
delete
My guess is it's a timing issue during the delete and we're maybe not
waiting long enough in tempest? Unless something failed when trying to
delete port 3080b6d8-f622-496b-a749-0b7def78fb6c?
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Body%3A%5C%22%20AND%20message%3A%5C%22%3A%20One%20or%20more%20ports%20have%20an%20IP%20allocation%20from%20this%20subnet%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d
15 hits in 7 days, check and gate, all failures.
I want to say I've seen this before in the gate but couldn't find an
existing bug for it, unless it was already fixed once and this is a
regression.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1813959/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1639955] Re: bad test for snappy systems
This issue was fixed a while back in:
https://github.com/canonical/cloud-init/commit/4bcc94730
cloud-init retains the legacy bits, but prefers the use of os-release
file or kernel commandline to for checking for UbuntuCore.
** Changed in: cloud-init (Ubuntu)
Status: Confirmed => Fix Released
** Changed in: cloud-init
Status: Confirmed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1639955
Title:
bad test for snappy systems
Status in cloud-init:
Fix Released
Status in cloud-init package in Ubuntu:
Fix Released
Bug description:
Reviewing the latest SRU for cloud-init, I noticed the following:
def system_is_snappy():
# channel.ini is configparser loadable.
# snappy will move to using /etc/system-image/config.d/*.ini
# this is certainly not a perfect test, but good enough for now.
content = load_file("/etc/system-image/channel.ini", quiet=True)
if 'ubuntu-core' in content.lower():
return True
if os.path.isdir("/etc/system-image/config.d/"):
return True
return False
This isn't a good test for whether a system is an ubuntu-core system.
'system-image' is historical baggage, and not likely to be present at
all in future versions.
I'm afraid I don't know a good alternative test offhand, but wanted to
log the bug so someone could look into it rather than being caught by
surprise when ubuntu-core image contents later change.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1639955/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1879407] [NEW] [OVN] Modifying FIP that is no associated causes ovn_revision_numbers to go stale
Public bug reported:
NOTE: This is a low priority issue, mostly because it eventually gets
fixed by maintenance task
CheckRevisionNumberCommand relies in finding a corresponding entry in OVN's NAT
table
in order to update the OVN_REV_NUM_EXT_ID_KEY to keep ovn and neutron databases
in sync.
Ref: http://lucasgom.es/posts/neutron_ovn_database_consistency.html
Trouble is that unless the floating ip is associated, there will be no
entries in OVN's NAT table, causing the call to
db_rev.bump_revision(context, floatingip, ovn_const.TYPE_FLOATINGIPS)
to not take place.
Steps to reproduce it:
# create a floating ip but do not associate it with anything so router_id is
None
FIP=172.24.4.8
openstack floating ip create --floating-ip-address ${FIP} public
FIP_UUID=$(openstack floating ip show ${FIP} -f value -c id) ; echo $FIP_UUID
# Mess with its name, which will bump revision on fip object
openstack floating ip set --description foo ${FIP_UUID}
Code when there is no NAT for a given FIP makes line 1044 skip line 1045
https://github.com/openstack/neutron/blob/15088b39bab715e40d8161a85c95ca400708c83f/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L1044
check_rev_cmd.result is None
The dbs are now the inconsistent state
mysql> use neutron;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from standardattributes where resource_type="floatingips";
++---+-+-+-+-+
| id | resource_type | created_at | updated_at | description
| revision_number |
++---+-+-+-+-+
| 49 | floatingips | 2020-05-18 20:56:51 | 2020-05-18 20:58:58 | foo2
| 2 |
++---+-+-+-+-+
1 row in set (0.01 sec)
mysql> select * from ovn_revision_numbers where resource_type="floatingips";
+--+--+---+-+-+-+
| standard_attr_id | resource_uuid| resource_type |
revision_number | created_at | updated_at |
+--+--+---+-+-+-+
| 49 | 5a1e1ffa-0312-4e78-b7a0-551c396bcf6b | floatingips |
0 | 2020-05-18 20:56:51 | 2020-05-18 20:57:08 |
+--+--+---+-+-+-+
1 row in set (0.00 sec)
Maintenance task fixes it up later
May 18 21:50:29 stack neutron-server[909]: DEBUG futurist.periodics [None
req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Submitting periodic
callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBIn\
consistenciesPeriodics.check_for_inconsistencies' {{(pid=3186)
_process_scheduled
/usr/local/lib/python3.6/dist-packages/futurist/periodics.py:642}}
May 18 21:50:29 stack neutron-server[909]: DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None
req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task:
Synchronizing Neutron and OVN datab\
ases {{(pid=3186) check_for_inconsistencies
/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:347}}
May 18 21:50:29 stack neutron-server[909]: DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None
req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Number of
inconsistencies found at \
create/update: floatingips=1 {{(pid=3186) _log
/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:325}}
May 18 21:50:29 stack neutron-server[909]: DEBUG
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None
req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Fixing
resource 6b876a35-d286-4407-\
b538-9ce07ab1a281 (type: floatingips) at create/update {{(pid=3186)
check_for_inconsistencies
/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:359}}
May 18 21:50:29 stack neutron-server[909]: INFO
neutron.db.ovn_revision_numbers_db [None
req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Successfully bumped
revision number for resource 6b876a35-d286-4407-b538-9ce07ab1\
a281 (type: floatingips) to 1
May 18 21:50:29 stack neutron-server[909]: INFO
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None
req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task:
Synchronization finished (took 0.08 \
seconds)
** Affects: neutron
Importance: Undecided
Status: New
** Description changed:
NOTE: This is a low priority issue, m
[Yahoo-eng-team] [Bug 1879403] [NEW] cc_chef does not support chef_license key to accept the chef EULA license
Public bug reported: This is a request to add support for the client configuration option "chef_license" in `chef_client.rb.tmpl` and the `chef` configuration block. Use Case: Latest version of chef products needs us to accept the EULA license agreement without accepting the license chef infra client run will not be initiated or chef registration to chef-server does not happen. As per chef documentation https://docs.chef.io/chef_license_accept/#chef-infra-client users can specify chef_license 'accept' in their Chef Infra Client and Chef Infra Server config. Example: # cloud-init chef: chef_license: "accept" install_type: "packages" server_url: https://api.opscode.com/organizations/myorg environment: dev validation_name: dev-validator validation_cert: dev-validator.pem run_list: role[db] encrypted_data_bag_secret: /etc/chef/encrypted_data_bag_secret => # /etc/chef/client.rb chef_license "accept" log_level :info log_location "/var/log/chef/client.log" ssl_verify_mode :verify_none validation_client_name "dev-validator" validation_key "/etc/chef/validation.pem" client_key "/etc/chef/client.pem" chef_server_url "https://api.opscode.com/organizations/myorg"; environment "dev" node_name "f3e2342498-8832492792-827349872492" json_attribs "/etc/chef/firstboot.json" file_cache_path "/var/cache/chef" file_backup_path "/var/backups/chef" pid_file "/var/run/chef/client.pid" Chef::Log::Formatter.show_time = true encrypted_data_bag_secret: /etc/chef/encrypted_data_bag_secret => Without this cc_chef does not work with chef-15 and above. ** Affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1879403 Title: cc_chef does not support chef_license key to accept the chef EULA license Status in cloud-init: New Bug description: This is a request to add support for the client configuration option "chef_license" in `chef_client.rb.tmpl` and the `chef` configuration block. Use Case: Latest version of chef products needs us to accept the EULA license agreement without accepting the license chef infra client run will not be initiated or chef registration to chef-server does not happen. As per chef documentation https://docs.chef.io/chef_license_accept/#chef-infra-client users can specify chef_license 'accept' in their Chef Infra Client and Chef Infra Server config. Example: # cloud-init chef: chef_license: "accept" install_type: "packages" server_url: https://api.opscode.com/organizations/myorg environment: dev validation_name: dev-validator validation_cert: dev-validator.pem run_list: role[db] encrypted_data_bag_secret: /etc/chef/encrypted_data_bag_secret => # /etc/chef/client.rb chef_license "accept" log_level :info log_location "/var/log/chef/client.log" ssl_verify_mode :verify_none validation_client_name "dev-validator" validation_key "/etc/chef/validation.pem" client_key "/etc/chef/client.pem" chef_server_url "https://api.opscode.com/organizations/myorg"; environment "dev" node_name "f3e2342498-8832492792-827349872492" json_attribs "/etc/chef/firstboot.json" file_cache_path "/var/cache/chef" file_backup_path "/var/backups/chef" pid_file "/var/run/chef/client.pid" Chef::Log::Formatter.show_time = true encrypted_data_bag_secret: /etc/chef/encrypted_data_bag_secret => Without this cc_chef does not work with chef-15 and above. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1879403/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1878979] Re: Quota code does not respect [api]/instance_list_per_project_cells
Reviewed: https://review.opendev.org/728575 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ab16946885e68ccdef7222a71cc0ad6f92b10de7 Submitter: Zuul Branch:master commit ab16946885e68ccdef7222a71cc0ad6f92b10de7 Author: Mohammed Naser Date: Fri May 15 16:49:12 2020 -0400 Make quotas respect instance_list_per_project_cells This option was introduced in order to limit queries to cells which only belong to the project we're interacting with. However, The nova quota code was not respecting it and therefore it was always trying to get the cells assigned to a project even with that option disabled. This patch makes the quota code respect that option and adds testing to ensure that enabling the option does make sure it doesn't search all cells but only the ones for this project. Closes-Bug: #1878979 Change-Id: I2e0d48e799e70d550f912ad8a424c86df3ade3a2 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1878979 Title: Quota code does not respect [api]/instance_list_per_project_cells Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: New Status in OpenStack Compute (nova) ussuri series: New Bug description: The function which counts resources using the legacy method involves getting a list of all cell mappings assigned to a specific project: https://github.com/openstack/nova/blob/575a91ff5be79ac35aef4b61d84c78c693693304/nova/quota.py#L1170-L1209 This code can be very heavy on a database which contains a lot of instances (but not a lot of mappings), potentially scanning millions of rows to gather 1-2 cell mappings. In a single cell environment, it is just extra CPU usage with exactly the same outcome. The [api]/instance_list_per_project_cells was introduced to workaround this: https://github.com/openstack/nova/blob/575a91ff5be79ac35aef4b61d84c78c693693304/nova/compute/instance_list.py#L146-L153 However, the quota code does not implement it which means quota count take a big toll on the database server. We should ideally mirror the same behaviour in the quota code. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1878979/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1878583] Re: Unable to createImage/snapshot paused volume backed instances
Reviewed: https://review.opendev.org/728011
Committed:
https://git.openstack.org/cgit/openstack/nova/commit/?id=cfde53e4b402e71d7f53b6e0ab232854dba160dc
Submitter: Zuul
Branch:master
commit cfde53e4b402e71d7f53b6e0ab232854dba160dc
Author: Lee Yarwood
Date: Thu May 14 10:49:07 2020 +0100
compute: Allow snapshots to be created from PAUSED volume backed instances
Iabeb44f843c3c04f767c4103038fcf6c52966ff3 allowed snapshots to be
created from PAUSED non-volume backed instances but missed the volume
backed use case.
This change simply adds PAUSED to the list of acceptable vm_states when
creating a snapshot from a volume backed instance in addition to the
already supported ACTIVE, STOPPED and SUSPENDED vm_states.
Closes-Bug: #1878583
Change-Id: I9f95a054de9d43ecaa50ff7ffc9343490e212d53
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1878583
Title:
Unable to createImage/snapshot paused volume backed instances
Status in OpenStack Compute (nova):
Fix Released
Bug description:
Description
===
Unable to createImage/snapshot paused volume backed instances.
Steps to reproduce
==
- Pause a volume backed instance
- Attempt to snapshot the instance using the createImage API
Expected result
===
A snapshot image is successfully created as is the case for paused instances
that are not volume backed.
Actual result
=
n-api returns the following error:
{'code': 409, 'message': "Cannot 'createImage' instance
bc5a7ae4-fca9-4d83-b1b8-5534f51a9404 while it is in vm_state paused"}
Environment
===
1. Exact version of OpenStack you are running. See the following
list for all releases: http://docs.openstack.org/releases/
master
2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?
N/A
2. Which storage type did you use?
(For example: Ceph, LVM, GPFS, ...)
What's the version of that?
N/A
3. Which networking type did you use?
(For example: nova-network, Neutron with OpenVSwitch, ...)
N/A
Logs & Configs
==
As above.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1878583/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1879356] [NEW] `cloud-init devel schema --annotate` fails for integer keys which do not roundtrip through string representation
Public bug reported:
When using the new snap.commands schema (introduced in
https://github.com/canonical/cloud-init/pull/364), it's possible to
trigger a bug in our assertion code. Specifically, this file will fail
validation (correctly, because `123` is an integer and not a string):
#cloud-config
snap:
commands:
01: ["foo", 123]
And then traceback during annotation:
$ cloud-init devel schema -c foo.yaml --annotate
Traceback (most recent call last):
File "/home/daniel/dev/cloud-init/cloudinit/config/schema.py", line 217, in
validate_cloudconfig_file
validate_cloudconfig_schema(
File "/home/daniel/dev/cloud-init/cloudinit/config/schema.py", line 121, in
validate_cloudconfig_schema
raise SchemaValidationError(errors)
cloudinit.config.schema.SchemaValidationError: Cloud config schema errors:
snap.commands.1: ['foo', 123] is not valid under any of the given schemas
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/daniel/.virtualenvs/cloud-init/bin/cloud-init", line 11, in
load_entry_point('cloud-init', 'console_scripts', 'cloud-init')()
File "/home/daniel/dev/cloud-init/cloudinit/cmd/main.py", line 891, in main
retval = util.log_time(
File "/home/daniel/dev/cloud-init/cloudinit/util.py", line 2618, in log_time
ret = func(*args, **kwargs)
File "/home/daniel/dev/cloud-init/cloudinit/config/schema.py", line 446, in
handle_schema_args
validate_cloudconfig_file(
File "/home/daniel/dev/cloud-init/cloudinit/config/schema.py", line 221, in
validate_cloudconfig_file
print(annotated_cloudconfig_file(
File "/home/daniel/dev/cloud-init/cloudinit/config/schema.py", line 153, in
annotated_cloudconfig_file
errors_by_line[schemapaths[path]].append(msg)
KeyError: 'snap.commands.1'
Note the `1` at the end of the key we're looking for (instead of 01).
If we modify the input file to drop the leading 0:
#cloud-config
snap:
commands:
1: ["foo", 123]
then we don't see a traceback:
$ cloud-init devel schema -c foo.yaml --annotate
#cloud-config
snap:
commands:
1: ["foo", 123] # E1
# Errors: -
# E1: ['foo', 123] is not valid under any of the given schemas
** Affects: cloud-init
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1879356
Title:
`cloud-init devel schema --annotate` fails for integer keys which do
not roundtrip through string representation
Status in cloud-init:
New
Bug description:
When using the new snap.commands schema (introduced in
https://github.com/canonical/cloud-init/pull/364), it's possible to
trigger a bug in our assertion code. Specifically, this file will
fail validation (correctly, because `123` is an integer and not a
string):
#cloud-config
snap:
commands:
01: ["foo", 123]
And then traceback during annotation:
$ cloud-init devel schema -c foo.yaml --annotate
Traceback (most recent call last):
File "/home/daniel/dev/cloud-init/cloudinit/config/schema.py", line 217,
in validate_cloudconfig_file
validate_cloudconfig_schema(
File "/home/daniel/dev/cloud-init/cloudinit/config/schema.py", line 121,
in validate_cloudconfig_schema
raise SchemaValidationError(errors)
cloudinit.config.schema.SchemaValidationError: Cloud config schema errors:
snap.commands.1: ['foo', 123] is not valid under any of the given schemas
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/daniel/.virtualenvs/cloud-init/bin/cloud-init", line 11, in
load_entry_point('cloud-init', 'console_scripts', 'cloud-init')()
File "/home/daniel/dev/cloud-init/cloudinit/cmd/main.py", line 891, in
main
retval = util.log_time(
File "/home/daniel/dev/cloud-init/cloudinit/util.py", line 2618, in
log_time
ret = func(*args, **kwargs)
File "/home/daniel/dev/cloud-init/cloudinit/config/schema.py", line 446,
in handle_schema_args
validate_cloudconfig_file(
File "/home/daniel/dev/cloud-init/cloudinit/config/schema.py", line 221,
in validate_cloudconfig_file
print(annotated_cloudconfig_file(
File "/home/daniel/dev/cloud-init/cloudinit/config/schema.py", line 153,
in annotated_cloudconfig_file
errors_by_line[schemapaths[path]].append(msg)
KeyError: 'snap.commands.1'
Note the `1` at the end of the key we're looking for (instead of 01).
If we modify the input file to drop the leading 0:
#cloud-config
snap:
commands:
1: ["foo", 123]
then we don't see a traceback:
$ cloud-init devel schema -c foo.yaml --annotate
#cloud-config
snap:
comm
[Yahoo-eng-team] [Bug 1879307] [NEW] Recent stable/rocky change breaks networking-calico's interface driver
Public bug reported: This merge - https://opendev.org/openstack/neutron/commit/a6fb2faaa5d46656db9085ad6bcfc65ded807871 - to the Neutron stable/rocky branch on April 23rd, has broken my team's Neutron plugin, by requiring 3rd party LinuxInterfaceDriver subclasses to take a new 'link_up' argument in their 'plug_new' method. Here's the fix that I've now made: https://github.com/projectcalico /networking- calico/pull/21/commits/bfd54aa841abbba4c591126b0dba083b93c84536 However, this likely affects other out-of-tree plugins as well, and there is still the likelihood of breakage if someone is running Rocky with an affected-but-unfixed plugin, and uses the latest Rocky code. Apparently there will not be another Rocky patch release, so I wonder if it would be better to revert the incompatible change? ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1879307 Title: Recent stable/rocky change breaks networking-calico's interface driver Status in neutron: New Bug description: This merge - https://opendev.org/openstack/neutron/commit/a6fb2faaa5d46656db9085ad6bcfc65ded807871 - to the Neutron stable/rocky branch on April 23rd, has broken my team's Neutron plugin, by requiring 3rd party LinuxInterfaceDriver subclasses to take a new 'link_up' argument in their 'plug_new' method. Here's the fix that I've now made: https://github.com/projectcalico /networking- calico/pull/21/commits/bfd54aa841abbba4c591126b0dba083b93c84536 However, this likely affects other out-of-tree plugins as well, and there is still the likelihood of breakage if someone is running Rocky with an affected-but-unfixed plugin, and uses the latest Rocky code. Apparently there will not be another Rocky patch release, so I wonder if it would be better to revert the incompatible change? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1879307/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1879301] [NEW] [OVN] networking-ovn-tempest-dsvm-ovs-release-python2 job starts to fail on tempest py2 installation
Public bug reported: The job networking-ovn-tempest-dsvm-ovs-release-python2 starts to fail on tempest py2 installation. Its blocking stable/train and maybe other stable branches. 2020-05-18 07:48:07.702856 | controller | Collecting oslo.concurrency===4.0.2 (from -c https://releases.openstack.org/constraints/upper/master (line 24)) 2020-05-18 07:48:07.702895 | controller | ERROR: Could not find a version that satisfies the requirement oslo.concurrency===4.0.2 (from -c https://releases.openstack.org/constraints/upper/master (line 24)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.7.0, 2.8.0, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, 3.13.0, 3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.17.0, 3.18.0, 3.18.1, 3.19.0, 3.20.0, 3.21.0, 3.21.1, 3.21.2, 3.22.0, 3.23.0, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.27.0, 3.28.0, 3.28.1, 3.29.0, 3.29.1, 3.30.0, 3.31.0) 2020-05-18 07:48:07.702935 | controller | ERROR: No matching distribution found for oslo.concurrency===4.0.2 (from -c https://releases.openstack.org/constraints/upper/master (line 24)) 2020-05-18 07:48:07.702962 | controller | WARNING: You are using pip version 19.2.3, however version 20.1 is available. 2020-05-18 07:48:07.702983 | controller | You should consider upgrading via the 'pip install --upgrade pip' command. 2020-05-18 07:48:07.703003 | controller | 2020-05-18 07:48:07.703227 | controller | === log end 2020-05-18 07:48:07.703702 | controller | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1) https://3ba3378426f3a529e977- d1da58634df71c1c590b1ad3c3dea539.ssl.cf5.rackcdn.com/715447/10/check/networking-ovn-tempest-dsvm-ovs-release-python2/0474be3/job-output.txt ** Affects: neutron Importance: High Assignee: Maciej Jozefczyk (maciej.jozefczyk) Status: Confirmed ** Tags: ovn ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1879301 Title: [OVN] networking-ovn-tempest-dsvm-ovs-release-python2 job starts to fail on tempest py2 installation Status in neutron: Confirmed Bug description: The job networking-ovn-tempest-dsvm-ovs-release-python2 starts to fail on tempest py2 installation. Its blocking stable/train and maybe other stable branches. 2020-05-18 07:48:07.702856 | controller | Collecting oslo.concurrency===4.0.2 (from -c https://releases.openstack.org/constraints/upper/master (line 24)) 2020-05-18 07:48:07.702895 | controller | ERROR: Could not find a version that satisfies the requirement oslo.concurrency===4.0.2 (from -c https://releases.openstack.org/constraints/upper/master (line 24)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.7.0, 2.8.0, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, 3.13.0, 3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.17.0, 3.18.0, 3.18.1, 3.19.0, 3.20.0, 3.21.0, 3.21.1, 3.21.2, 3.22.0, 3.23.0, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.27.0, 3.28.0, 3.28.1, 3.29.0, 3.29.1, 3.30.0, 3.31.0) 2020-05-18 07:48:07.702935 | controller | ERROR: No matching distribution found for oslo.concurrency===4.0.2 (from -c https://releases.openstack.org/constraints/upper/master (line 24)) 2020-05-18 07:48:07.702962 | controller | WARNING: You are using pip version 19.2.3, however version 20.1 is available. 2020-05-18 07:48:07.702983 | controller | You should consider upgrading via the 'pip install --upgrade pip' command. 2020-05-18 07:48:07.703003 | controller | 2020-05-18 07:48:07.703227 | controller | === log end 2020-05-18 07:48:07.703702 | controller | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1) https://3ba3378426f3a529e977- d1da58634df71c1c590b1ad3c3dea539.ssl.cf5.rackcdn.com/715447/10/check/networking-ovn-tempest-dsvm-ovs-release-python2/0474be3/job-output.txt To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1879301/+subscriptio
[Yahoo-eng-team] [Bug 1877560] Re: Optimize "QosPolicy" OVO bound objects retrieve methods
Reviewed: https://review.opendev.org/726358 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1f018514d71daab6ff3d4bd508318ce8ed8d1045 Submitter: Zuul Branch:master commit 1f018514d71daab6ff3d4bd508318ce8ed8d1045 Author: Rodolfo Alonso Hernandez Date: Fri May 8 11:32:17 2020 + Optimize QoS bound objects queries Optimize the following methods: - get_bound_networks - get_bound_ports - get_bound_floatingips - get_bound_routers Those methods, using the "QosPolicy__Binding" OVO interface, were retrieving all objects with a certain policy ID and then returning only the ID of the __ object. That means to retrieve a full register list from the DB, then converted them to OVOs and extract only the __ ID. This patch retrieves only the __ object IDs from the DB and returns the list, without the OVO conversion. Change-Id: I891eba93b3b4abaec8ada13a032b5440cbb0548d Closes-Bug: #1877560 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1877560 Title: Optimize "QosPolicy" OVO bound objects retrieve methods Status in neutron: Fix Released Bug description: The following methods are inefficient: - get_bound_networks - get_bound_ports - get_bound_floatingips - get_bound_routers Those methods, using the "QosPolicy__Binding" OVO interface, are retrieving all objects with a certain policy ID and then returning only the ID of the __ object. That means we retrieve a full register list from the DB, then those registers are converted to OVO and then we extract only the __ ID. This is obviously inefficient, regardless of the simplicity of the "QosPolicy__Binding" OVO and the 1:1 parity to the DB associated register. We should instead only retrieve the __ object ID from the DB and return it. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1877560/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1879294] [NEW] Support fw_cfg data source
Public bug reported: It would be awesome if cloud-init supported pulling config from QEMU's fw_cfg. There has been kernel support for it since around 4.15 (IIRC), so it should be widely available at this point. ** Affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1879294 Title: Support fw_cfg data source Status in cloud-init: New Bug description: It would be awesome if cloud-init supported pulling config from QEMU's fw_cfg. There has been kernel support for it since around 4.15 (IIRC), so it should be widely available at this point. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1879294/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1869050] Re: migration of anti-affinity server fails due to stale scheduler instance info
** Also affects: nova/ussuri Importance: Undecided Status: New ** Changed in: nova/ussuri Importance: Undecided => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1869050 Title: migration of anti-affinity server fails due to stale scheduler instance info Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Compute (nova) queens series: Triaged Status in OpenStack Compute (nova) rocky series: Triaged Status in OpenStack Compute (nova) stein series: Triaged Status in OpenStack Compute (nova) train series: Triaged Status in OpenStack Compute (nova) ussuri series: In Progress Bug description: Description === Steps to reproduce == Have a deployment with 3 compute nodes * make sure that the deployment is configured with tracks_instance_changes=True (True is the default) * create and server group with anti-affinity policy * boot server1 into the group * boot server2 into the group * migrate server2 * confirm the migration * boot server3 Make sure that between the last two steps there was no periodic _sync_scheduler_instance_info running on the compute that was hosted server2 before the migration. This could done by doing the last too steps after each other without waiting too much as interval of that periodic (scheduler_instance_sync_interval) is defaulted to 120 sec. Expected result === server3 is booted on the host where server2 is moved away Actual result = server3 cannot be booted (NoValidHost) Triage == The confirm resize call on the source compute does not update the scheduler that the instance is removed from this host. This makes the scheduler instance info stale and causing the subsequent scheduling error. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1869050/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
