[Yahoo-eng-team] [Bug 1920256] [NEW] Typo in sort dir parameter descriptions
Public bug reported: This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes: - [x] This doc is inaccurate in this way: There is a typo in the server and flavor sort_dir parameter descriptions, you can see it here: https://github.com/openstack/nova/search?q=%22direction+of+the+direction+of+the%22 Fix: s/direction of the direction of the/direction of the/ --- Release: on 2020-03-11 10:38:56 SHA: 1f125b97c663968c600f8898e4d42f11d50af83a Source: https://opendev.org/openstack/nova/src/api-ref/source/index.rst URL: https://docs.openstack.org/api-ref/compute/ ** Affects: nova Importance: Undecided Status: New ** Tags: api-ref -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1920256 Title: Typo in sort dir parameter descriptions Status in OpenStack Compute (nova): New Bug description: This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes: - [x] This doc is inaccurate in this way: There is a typo in the server and flavor sort_dir parameter descriptions, you can see it here: https://github.com/openstack/nova/search?q=%22direction+of+the+direction+of+the%22 Fix: s/direction of the direction of the/direction of the/ --- Release: on 2020-03-11 10:38:56 SHA: 1f125b97c663968c600f8898e4d42f11d50af83a Source: https://opendev.org/openstack/nova/src/api-ref/source/index.rst URL: https://docs.openstack.org/api-ref/compute/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1920256/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1893238] [NEW] nova document search result links not working
Public bug reported: The resulting links from the nova document search aren't working. For example, searching for cross_az_attach here: https://docs.openstack.org/nova/latest/search.html?q=cross_az_attach Yields a few links but clicking on them gives a 404, e.g.: https://docs.openstack.org/nova/latest/admin/availability- zonesundefined?highlight=cross_az_attach If you replace that "undefined" with ".html" it works. ** Affects: nova Importance: Undecided Status: Confirmed ** Tags: doc -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1893238 Title: nova document search result links not working Status in OpenStack Compute (nova): Confirmed Bug description: The resulting links from the nova document search aren't working. For example, searching for cross_az_attach here: https://docs.openstack.org/nova/latest/search.html?q=cross_az_attach Yields a few links but clicking on them gives a 404, e.g.: https://docs.openstack.org/nova/latest/admin/availability- zonesundefined?highlight=cross_az_attach If you replace that "undefined" with ".html" it works. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1893238/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1866106] Re: Can't set "pointer_model = None" in nova.conf
** Changed in: oslo.config
Status: New => Won't Fix
** Tags added: config libvirt
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1866106
Title:
Can't set "pointer_model = None" in nova.conf
Status in OpenStack Compute (nova):
In Progress
Status in oslo.config:
Won't Fix
Bug description:
Description
===
nova.conf includes option pointer_model. The help text in the config
has 2 "possible values" sections (copied below) specifying either
"None" or "" as correct values. Neither of these is accepted by
Nova. Here are the error messages from nova-compute.log:
2020-03-03 11:05:17.233 228915 ERROR nova ConfigFileValueError: Value
for option pointer_model is not valid: Valid values are [None,
ps2mouse, usbtablet], but found ''
2020-03-03 11:06:24.761 229290 ERROR nova ConfigFileValueError: Value
for option pointer_model is not valid: Valid values are [None,
ps2mouse, usbtablet], but found 'None'
#
# Generic property to specify the pointer type.
#
# Input devices allow interaction with a graphical framebuffer. For
# example to provide a graphic tablet for absolute cursor movement.
#
# If set, the 'hw_pointer_model' image property takes precedence over
# this configuration option.
#
# Possible values:
#
# * None: Uses default behavior provided by drivers (mouse on PS2 for
# libvirt x86)
# * ps2mouse: Uses relative movement. Mouse connected by PS2
# * usbtablet: Uses absolute movement. Tablet connect by USB
#
# Related options:
#
# * usbtablet must be configured with VNC enabled or SPICE enabled and SPICE
# agent disabled. When used with libvirt the instance mode should be
# configured as HVM.
# (string value)
# Possible values:
# -
# ps2mouse -
# usbtablet -
#pointer_model = usbtablet
Steps to reproduce
==
On an openstack hypervisor:
1. Edit nova.conf and change line "#pointer_model = usbtablet" to either
"pointer_model = None" or "pointer_model = "
2. Restart nova-compute service
3. Tail nova-compute.log
Expected result
===
Nova runs without errors and does not load the USB driver.
Actual result
=
Nova throws the error described above.
Environment
===
1. Openstack version is Rocky:
root@us01odc-p01-hv227:~# dpkg -l | grep nova
ii nova-common 2:18.2.1-0ubuntu1~cloud4
all OpenStack Compute - common files
ii nova-compute 2:18.2.1-0ubuntu1~cloud4
all OpenStack Compute - compute node base
ii nova-compute-kvm 2:18.2.1-0ubuntu1~cloud4
all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:18.2.1-0ubuntu1~cloud4
all OpenStack Compute - compute node libvirt support
ii python-nova 2:18.2.1-0ubuntu1~cloud4
all OpenStack Compute Python 2 libraries
ii python-novaclient 2:11.0.0-0ubuntu1~cloud0
all client library for OpenStack Compute API - Python 2.7
2. Hypervisor: libvirt+KVM
root@us01odc-p01-hv227:~# libvirtd --version
libvirtd (libvirt) 4.0.0
root@us01odc-p01-hv227:~# kvm --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.21)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
3. Storage type: local LVM
root@us01odc-p01-hv227:~# lvm version
LVM version: 2.02.176(2) (2017-11-03)
Library version: 1.02.145 (2017-11-03)
Driver version: 4.39.0
Configuration: ./configure --build=x86_64-linux-gnu --prefix=/usr
--includedir=${prefix}/include --mandir=${prefix}/share/man
--infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var
--disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu
--libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run
--disable-maintainer-mode --disable-dependency-tracking --exec-prefix=
--bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin
--with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2
--with-cache=internal --with-clvmd=corosync --with-cluster=internal
--with-device-uid=0 --with-device-gid=6 --with-device-mode=0660
--with-default-pid-dir=/run --with-default-run-dir=/run/lvm
--with-default-locking-dir=/run/lock/lvm --with-thin=internal
--with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump
--with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping
--enable-cmdlib --enable-cmirrord --enable-dmeventd --enable-dbus-service
--enable-lvmetad --enable-lvmlockd-dlm --enable-lvmlockd-sanlock
--enable-lvmpolld --enable-notify-dbus
[Yahoo-eng-team] [Bug 1866106] Re: Can't set "pointer_model = None" in nova.conf
The problem is in oslo.config I think right here:
https://github.com/openstack/oslo.config/blob/20a7cee3e3019d60c4b367bb76922a1db41d1750/oslo_config/types.py#L142
That's coercing the value None to a string 'None' so it fails. According
to Ben Nemec:
(02:13:48 PM) bnemec: The only way for a config opt to have a None value is for
that to be the default and for the opt to be unset.
(02:14:14 PM) bnemec: So completely absent from the file, not something like
"opt="
But that seems like a bug because I would think that code could be
smarter about not coercing the value if the value is None, None is a
valid choice and a default is set (so you need to override the default).
** Also affects: oslo.config
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1866106
Title:
Can't set "pointer_model = None" in nova.conf
Status in OpenStack Compute (nova):
Confirmed
Status in oslo.config:
New
Bug description:
Description
===
nova.conf includes option pointer_model. The help text in the config
has 2 "possible values" sections (copied below) specifying either
"None" or "" as correct values. Neither of these is accepted by
Nova. Here are the error messages from nova-compute.log:
2020-03-03 11:05:17.233 228915 ERROR nova ConfigFileValueError: Value
for option pointer_model is not valid: Valid values are [None,
ps2mouse, usbtablet], but found ''
2020-03-03 11:06:24.761 229290 ERROR nova ConfigFileValueError: Value
for option pointer_model is not valid: Valid values are [None,
ps2mouse, usbtablet], but found 'None'
#
# Generic property to specify the pointer type.
#
# Input devices allow interaction with a graphical framebuffer. For
# example to provide a graphic tablet for absolute cursor movement.
#
# If set, the 'hw_pointer_model' image property takes precedence over
# this configuration option.
#
# Possible values:
#
# * None: Uses default behavior provided by drivers (mouse on PS2 for
# libvirt x86)
# * ps2mouse: Uses relative movement. Mouse connected by PS2
# * usbtablet: Uses absolute movement. Tablet connect by USB
#
# Related options:
#
# * usbtablet must be configured with VNC enabled or SPICE enabled and SPICE
# agent disabled. When used with libvirt the instance mode should be
# configured as HVM.
# (string value)
# Possible values:
# -
# ps2mouse -
# usbtablet -
#pointer_model = usbtablet
Steps to reproduce
==
On an openstack hypervisor:
1. Edit nova.conf and change line "#pointer_model = usbtablet" to either
"pointer_model = None" or "pointer_model = "
2. Restart nova-compute service
3. Tail nova-compute.log
Expected result
===
Nova runs without errors and does not load the USB driver.
Actual result
=
Nova throws the error described above.
Environment
===
1. Openstack version is Rocky:
root@us01odc-p01-hv227:~# dpkg -l | grep nova
ii nova-common 2:18.2.1-0ubuntu1~cloud4
all OpenStack Compute - common files
ii nova-compute 2:18.2.1-0ubuntu1~cloud4
all OpenStack Compute - compute node base
ii nova-compute-kvm 2:18.2.1-0ubuntu1~cloud4
all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:18.2.1-0ubuntu1~cloud4
all OpenStack Compute - compute node libvirt support
ii python-nova 2:18.2.1-0ubuntu1~cloud4
all OpenStack Compute Python 2 libraries
ii python-novaclient 2:11.0.0-0ubuntu1~cloud0
all client library for OpenStack Compute API - Python 2.7
2. Hypervisor: libvirt+KVM
root@us01odc-p01-hv227:~# libvirtd --version
libvirtd (libvirt) 4.0.0
root@us01odc-p01-hv227:~# kvm --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.21)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
3. Storage type: local LVM
root@us01odc-p01-hv227:~# lvm version
LVM version: 2.02.176(2) (2017-11-03)
Library version: 1.02.145 (2017-11-03)
Driver version: 4.39.0
Configuration: ./configure --build=x86_64-linux-gnu --prefix=/usr
--includedir=${prefix}/include --mandir=${prefix}/share/man
--infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var
--disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu
--libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run
--disable-maintainer-mode --disable-dependency-tracking --exec-prefix=
--bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin
[Yahoo-eng-team] [Bug 1856925] Re: Nova compute service exception that performs cold migration virtual machine stuck in resize state.
** Also affects: nova/train Importance: Undecided Status: New ** Changed in: nova/train Status: New => In Progress ** Changed in: nova/train Importance: Undecided => Low ** Changed in: nova/train Assignee: (unassigned) => Lee Yarwood (lyarwood) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1856925 Title: Nova compute service exception that performs cold migration virtual machine stuck in resize state. Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) train series: In Progress Bug description: Description: In the case of a nova-compute service exception, such as down, the instance gets stuck in the resize state during cold migration and cannot perform evacuation.The command request for nova API is also issued, server_status and Task State have been changed, but compute cannot receive the request, resulting in the server State remaining in the resize State. When nova-compute is restarted, the server State becomes ERROR.It is recommended to add validation to prevent instances from entering inoperable states. This can also happen with commands such as stop/rebuild/reboot. Environment: 1. openstack-Q;nova -version:9.1.1 2. hypervisor: Libvirt + KVM 3. One control node, two compute nodes. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1856925/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1858667] [NEW] TestMultiCellMigrate.test_poll_unconfirmed_resizes_with_(no_)upcall race failing with greenlet error: Error auto-confirming resize: Reader
Public bug reported:
Seen here:
https://35ea41893426ced4e7b7-c7b305d9ddba843cc11c9f16befc26b2.ssl.cf1.rackcdn.com/699291/4/check
/nova-tox-functional-py36/6f68741/testr_results.html.gz
ft1.12:
nova.tests.functional.test_cross_cell_migrate.TestMultiCellMigrate.test_poll_unconfirmed_resizes_with_upcalltesttools.testresult.real._StringException:
pythonlogging:'': {{{
2020-01-07 12:39:14,920 WARNING [placement.db_api] TransactionFactory already
started, not reconfiguring.
2020-01-07 12:39:15,344 INFO [nova.service] Starting conductor node (version
20.1.0)
2020-01-07 12:39:15,379 INFO [nova.service] Starting scheduler node (version
20.1.0)
2020-01-07 12:39:15,425 INFO [nova.virt.driver] Loading compute driver
'fake.MediumFakeDriver'
2020-01-07 12:39:15,426 INFO [nova.service] Starting compute node (version
20.1.0)
2020-01-07 12:39:15,443 WARNING [nova.compute.manager] Compute node host1 not
found in the database. If this is the first time this service is starting on
this host, then you can ignore this warning.
2020-01-07 12:39:15,447 INFO [nova.compute.manager] Looking for unclaimed
instances stuck in BUILDING status for nodes managed by this host
2020-01-07 12:39:15,457 WARNING [nova.compute.manager] No compute node record
found for host host1. If this is the first time this service is starting on
this host, then you can ignore this warning.
2020-01-07 12:39:15,462 WARNING [nova.compute.resource_tracker] No compute node
record for host1:host1
2020-01-07 12:39:15,466 INFO [nova.compute.resource_tracker] Compute node
record created for host1:host1 with uuid: 30117184-3985-48b6-8e7e-5611c45c112c
2020-01-07 12:39:15,509 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/resource_providers?in_tree=30117184-3985-48b6-8e7e-5611c45c112c"
status: 200 len: 26 microversion: 1.14
2020-01-07 12:39:15,518 INFO [placement.requestlog] 127.0.0.1 "POST
/placement/resource_providers" status: 200 len: 828 microversion: 1.20
2020-01-07 12:39:15,519 INFO [nova.scheduler.client.report]
[req-6c8c5604-6d3d-49b4-ae66-3aadfeb1ff27] Created resource provider record via
placement API for resource provider with UUID
30117184-3985-48b6-8e7e-5611c45c112c and name host1.
2020-01-07 12:39:15,536 INFO [placement.requestlog] 127.0.0.1 "PUT
/placement/resource_providers/30117184-3985-48b6-8e7e-5611c45c112c/inventories"
status: 200 len: 401 microversion: 1.26
2020-01-07 12:39:15,545 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/traits?name=in:COMPUTE_VOLUME_MULTI_ATTACH,COMPUTE_TRUSTED_CERTS,COMPUTE_DEVICE_TAGGING,COMPUTE_VOLUME_EXTEND,COMPUTE_NODE,COMPUTE_NET_ATTACH_INTERFACE_WITH_TAG,COMPUTE_NET_ATTACH_INTERFACE,COMPUTE_VOLUME_ATTACH_WITH_TAG,COMPUTE_IMAGE_TYPE_RAW"
status: 200 len: 268 microversion: 1.6
2020-01-07 12:39:15,561 INFO [placement.requestlog] 127.0.0.1 "PUT
/placement/resource_providers/30117184-3985-48b6-8e7e-5611c45c112c/traits"
status: 200 len: 303 microversion: 1.6
2020-01-07 12:39:15,589 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST
/v2.1/os-aggregates" status: 200 len: 221 microversion: 2.81 time: 0.021327
2020-01-07 12:39:15,634 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/resource_providers?name=host1" status: 200 len: 432 microversion: 1.0
2020-01-07 12:39:15,643 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/resource_providers/30117184-3985-48b6-8e7e-5611c45c112c/aggregates"
status: 200 len: 53 microversion: 1.19
2020-01-07 12:39:15,655 INFO [placement.requestlog] 127.0.0.1 "PUT
/placement/resource_providers/30117184-3985-48b6-8e7e-5611c45c112c/aggregates"
status: 200 len: 91 microversion: 1.19
2020-01-07 12:39:15,665 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST
/v2.1/os-aggregates/1/action" status: 200 len: 285 microversion: 2.81 time:
0.072452
2020-01-07 12:39:15,683 INFO [nova.virt.driver] Loading compute driver
'fake.MediumFakeDriver'
2020-01-07 12:39:15,684 INFO [nova.service] Starting compute node (version
20.1.0)
2020-01-07 12:39:15,703 WARNING [nova.compute.manager] Compute node host2 not
found in the database. If this is the first time this service is starting on
this host, then you can ignore this warning.
2020-01-07 12:39:15,708 INFO [nova.compute.manager] Looking for unclaimed
instances stuck in BUILDING status for nodes managed by this host
2020-01-07 12:39:15,718 WARNING [nova.compute.manager] No compute node record
found for host host2. If this is the first time this service is starting on
this host, then you can ignore this warning.
2020-01-07 12:39:15,723 WARNING [nova.compute.resource_tracker] No compute node
record for host2:host2
2020-01-07 12:39:15,732 INFO [nova.compute.resource_tracker] Compute node
record created for host2:host2 with uuid: 7e192b8c-5187-467c-8e7e-f8360fbce19e
2020-01-07 12:39:15,774 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/resource_providers?in_tree=7e192b8c-5187-467c-8e7e-f8360fbce19e"
status: 200 len: 26 microversion: 1.14
2020-01-07 12:39:15,782 INFO [placement.requestlog]
[Yahoo-eng-team] [Bug 1857306] Re: _bury_in_cell0 could not handle instance duplicate exception
Looks like the regression was introduced in Ocata: https://review.opendev.org/#/c/319379/ ** Changed in: nova Status: New => Triaged ** Changed in: nova Importance: Undecided => Low ** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/ocata Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/pike Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1857306 Title: _bury_in_cell0 could not handle instance duplicate exception Status in OpenStack Compute (nova): Triaged Status in OpenStack Compute (nova) ocata series: New Status in OpenStack Compute (nova) pike series: New Status in OpenStack Compute (nova) queens series: New Status in OpenStack Compute (nova) rocky series: New Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: New Bug description: For stable/stein if there were NoValidHost from scheduler, conductor should create Instance object in cell0. But I found if there is additional exception(InstanceExist) in the function, conductor could not catch the exception which result in instance state stuck in 'scheduling'. How to reproduce 1. osapi_compute_unique_server_name_scope = True 2. Cellv2 3. Create instance with hostname which previously built and without any valid host. Result 1. Conductor does not make instance DB creation in cell0 from _bury_in_cell0 What to expect 1. nova_cell0 create instance in cell0 with error state. Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1857306/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1844929] Re: grenade jobs failing due to "Timed out waiting for response from cell" in scheduler
** Also affects: grenade Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1844929 Title: grenade jobs failing due to "Timed out waiting for response from cell" in scheduler Status in grenade: New Status in OpenStack Compute (nova): Confirmed Bug description: Seen here: https://zuul.opendev.org/t/openstack/build/d53346210978403f888b85b82b2fe0c7/log/logs/screen-n-sch.txt.gz?severity=3#2368 Sep 22 00:50:54.174385 ubuntu-bionic-ovh-gra1-0011664420 nova- scheduler[18043]: WARNING nova.context [None req- 1929039e-1517-4326-9700-738d4b570ba6 tempest- AttachInterfacesUnderV243Test-2009753731 tempest- AttachInterfacesUnderV243Test-2009753731] Timed out waiting for response from cell 8acfb79b-2e40-4e1c-bc3d-d404dac6db90 Looks like something is causing timeouts reaching cell1 during grenade runs. The only errors I see in the rabbit logs are these for the uwsgi (API) servers: =ERROR REPORT 22-Sep-2019::00:35:30 === closing AMQP connection <0.1511.0> (217.182.141.188:48492 -> 217.182.141.188:5672 - uwsgi:19453:72e08501-61ca-4ade-865e- f0605979ed7d): missed heartbeats from client, timeout: 60s -- It looks like we don't have mysql logs in this grenade run, maybe we need a fix like this somewhere for grenade: https://github.com/openstack/devstack/commit/f92c346131db2c89b930b1a23f8489419a2217dc logstash shows 1101 hits in the last 7 days, since Sept 17 actually: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Timed%20out%20waiting%20for%20response%20from%20cell%5C%22%20AND%20tags%3A%5C%22screen-n-sch.txt%5C%22=7d check and gate queues, all failures. It also appears to only show up on fortnebula and OVH nodes, primarily fortnebula. I wonder if there is a performing/timing issue if those nodes are slower and we aren't waiting for something during the grenade upgrade before proceeding. To manage notifications about this bug go to: https://bugs.launchpad.net/grenade/+bug/1844929/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1857139] Re: TypeError: object of type 'object' has no len() from resources_from_request_spec when cells are down
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Exception%20during%20message%20handling%3A%20TypeError%3A%20object%20of%20type%20'object'%20has%20no%20len()%5C%22%20AND%20tags%3A%5C%22screen-n-sch.txt%5C%22=7d ** Also affects: nova/train Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1857139 Title: TypeError: object of type 'object' has no len() from resources_from_request_spec when cells are down Status in OpenStack Compute (nova): Triaged Status in OpenStack Compute (nova) train series: New Bug description: Seen here: https://zuul.opendev.org/t/openstack/build/c187e207bc1c48a0a7fa49ef9798b696/log/logs/screen-n-sch.txt.gz#2529 cell1 is down so the call to scatter_gather_cells in get_compute_nodes_by_host_or_node yields a result but it's not a ComputeNodeList, it's the did_not_respond_sentinel object: https://github.com/openstack/nova/blob/02019d2660dfce3facdd64ecdb2bd60ba4a91c6d/nova/scheduler/host_manager.py#L705 https://github.com/openstack/nova/blob/02019d2660dfce3facdd64ecdb2bd60ba4a91c6d/nova/context.py#L454 which results in an error here: https://github.com/openstack/nova/blob/02019d2660dfce3facdd64ecdb2bd60ba4a91c6d/nova/scheduler/utils.py#L612 The HostManager.get_compute_nodes_by_host_or_node method should filter out fail/timeout results from the scatter_gather_cells results. We'll get a NoValidHost either way but this is better than the traceback with the TypeError in it. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1857139/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1857139] [NEW] TypeError: object of type 'object' has no len() from resources_from_request_spec when cells are down
Public bug reported: Seen here: https://zuul.opendev.org/t/openstack/build/c187e207bc1c48a0a7fa49ef9798b696/log/logs/screen-n-sch.txt.gz#2529 cell1 is down so the call to scatter_gather_cells in get_compute_nodes_by_host_or_node yields a result but it's not a ComputeNodeList, it's the did_not_respond_sentinel object: https://github.com/openstack/nova/blob/02019d2660dfce3facdd64ecdb2bd60ba4a91c6d/nova/scheduler/host_manager.py#L705 https://github.com/openstack/nova/blob/02019d2660dfce3facdd64ecdb2bd60ba4a91c6d/nova/context.py#L454 which results in an error here: https://github.com/openstack/nova/blob/02019d2660dfce3facdd64ecdb2bd60ba4a91c6d/nova/scheduler/utils.py#L612 The HostManager.get_compute_nodes_by_host_or_node method should filter out fail/timeout results from the scatter_gather_cells results. We'll get a NoValidHost either way but this is better than the traceback with the TypeError in it. ** Affects: nova Importance: Low Assignee: Matt Riedemann (mriedem) Status: Triaged ** Affects: nova/train Importance: Undecided Status: New ** Tags: scheduler -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1857139 Title: TypeError: object of type 'object' has no len() from resources_from_request_spec when cells are down Status in OpenStack Compute (nova): Triaged Status in OpenStack Compute (nova) train series: New Bug description: Seen here: https://zuul.opendev.org/t/openstack/build/c187e207bc1c48a0a7fa49ef9798b696/log/logs/screen-n-sch.txt.gz#2529 cell1 is down so the call to scatter_gather_cells in get_compute_nodes_by_host_or_node yields a result but it's not a ComputeNodeList, it's the did_not_respond_sentinel object: https://github.com/openstack/nova/blob/02019d2660dfce3facdd64ecdb2bd60ba4a91c6d/nova/scheduler/host_manager.py#L705 https://github.com/openstack/nova/blob/02019d2660dfce3facdd64ecdb2bd60ba4a91c6d/nova/context.py#L454 which results in an error here: https://github.com/openstack/nova/blob/02019d2660dfce3facdd64ecdb2bd60ba4a91c6d/nova/scheduler/utils.py#L612 The HostManager.get_compute_nodes_by_host_or_node method should filter out fail/timeout results from the scatter_gather_cells results. We'll get a NoValidHost either way but this is better than the traceback with the TypeError in it. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1857139/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1857031] Re: cloud-init configures wrong interface when trying to configure two interfaces with OpenStack cloud
Marking invalid for nova since it sounds like this is a cloud-init issue. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1857031 Title: cloud-init configures wrong interface when trying to configure two interfaces with OpenStack cloud Status in cloud-init: Incomplete Status in OpenStack Compute (nova): Invalid Bug description: cloud-init 18.5: Node has 3 interfaces: -enp5s0f0 - not connected -enp5s0f1 - connected -ib0 - an HFI port Centos7.6 running on the node. Openstack boots the server with two interfaces enp5s0f1 and ib0 and it is successful but the node is not reachable. On the node, the cloud- init configures the wrong interface enp5s0f0. Please note that when I try to boot the server with only 1 interface enp5s0f1, everything works fine and the node is reachable too. Logs: http://paste.openstack.org/show/787707/ network-data and nics: http://paste.openstack.org/show/787797/ (note that enp5s0f1 is manually configured) To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1857031/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1835627] Re: test_shelve_unshelve_server failing on stable/pike (and probably ocata)
** Changed in: devstack Status: In Progress => Won't Fix ** Changed in: devstack-plugin-ceph Status: Confirmed => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1835627 Title: test_shelve_unshelve_server failing on stable/pike (and probably ocata) Status in devstack: Won't Fix Status in devstack-plugin-ceph: Invalid Status in OpenStack Compute (nova): Invalid Status in OpenStack Compute (nova) pike series: In Progress Bug description: Shelve tests should be run conditionally per branch as of this change: https://review.opendev.org/#/c/662327/ But it looks like on at least stable/pike the TARGET_BRANCH variable isn't set: http://logs.openstack.org/38/669538/1/check/devstack-plugin-ceph- tempest/a1dc2a3/controller/logs/devstacklog.txt.gz#_2019-07-05_22_04_10_952 2019-07-05 22:04:10.952 | ++ /opt/stack/devstack-plugin-ceph/devstack/plugin.sh:source:102 : [[ '' =~ stable/(ocata|pike|queens) ]] 2019-07-05 22:04:10.955 | ++ /opt/stack/devstack-plugin-ceph/devstack/plugin.sh:source:105 : iniset /opt/stack/tempest/etc/tempest.conf compute-feature-enabled shelve True To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1835627/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1835627] Re: test_shelve_unshelve_server failing on stable/pike (and probably ocata)
** Also affects: nova Importance: Undecided Status: New ** Also affects: nova/pike Importance: Undecided Status: New ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1835627 Title: test_shelve_unshelve_server failing on stable/pike (and probably ocata) Status in devstack: Won't Fix Status in devstack-plugin-ceph: Invalid Status in OpenStack Compute (nova): Invalid Status in OpenStack Compute (nova) pike series: In Progress Bug description: Shelve tests should be run conditionally per branch as of this change: https://review.opendev.org/#/c/662327/ But it looks like on at least stable/pike the TARGET_BRANCH variable isn't set: http://logs.openstack.org/38/669538/1/check/devstack-plugin-ceph- tempest/a1dc2a3/controller/logs/devstacklog.txt.gz#_2019-07-05_22_04_10_952 2019-07-05 22:04:10.952 | ++ /opt/stack/devstack-plugin-ceph/devstack/plugin.sh:source:102 : [[ '' =~ stable/(ocata|pike|queens) ]] 2019-07-05 22:04:10.955 | ++ /opt/stack/devstack-plugin-ceph/devstack/plugin.sh:source:105 : iniset /opt/stack/tempest/etc/tempest.conf compute-feature-enabled shelve True To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1835627/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1804502] Re: Rebuild server with NUMATopologyFilter enabled fails (in some cases)
** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Changed in: nova/train Status: New => In Progress ** Changed in: nova/queens Status: New => Triaged ** Changed in: nova/rocky Status: New => Triaged ** Changed in: nova/stein Status: New => Triaged ** Changed in: nova/queens Importance: Undecided => Low ** Changed in: nova/stein Importance: Undecided => Medium ** Changed in: nova/rocky Importance: Undecided => Medium ** Changed in: nova/train Assignee: (unassigned) => sean mooney (sean-k-mooney) ** Changed in: nova/train Importance: Undecided => Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1804502 Title: Rebuild server with NUMATopologyFilter enabled fails (in some cases) Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) queens series: Triaged Status in OpenStack Compute (nova) rocky series: Triaged Status in OpenStack Compute (nova) stein series: Triaged Status in OpenStack Compute (nova) train series: In Progress Bug description: Description === server rebuild will fail in nova scheduler on NUMATopologyFilter if the computes do not have enough capacity (even though clearly the running server is already accounted into that calculation) to resolve the issue a fix is required in NUMATopologyFilter to not perform the rebuild operation in the case that the request is due to rebuild. the result of such a case will be that server rebuild will fail with error of "no valid host found" (do not mix resize with rebuild functions...) Steps to reproduce == 1. create a flavor that contain metadata that will point to a specific compute (use host aggregate with same key:value metadata make sure flavor contain topology related metadata: hw:cpu_cores='1', hw:cpu_policy='dedicated', hw:cpu_sockets='6', hw:cpu_thread_policy='prefer', hw:cpu_threads='1', hw:mem_page_size='large', location='area51' 2. create a server on that compute (preferably using heat stack) 3. (try to) rebuild the server using stack update 4. issue reproduced Expected result === server in an active running state (if image was replaced in the rebuild command than with a reference to the new image in the server details. Actual result = server in error state with error of no valid host found. Message No valid host was found. There are not enough hosts available. Code 500 Details File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 966, in rebuild_instance return_alternates=False) File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 723, in _schedule_instances return_alternates=return_alternates) File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 907, in wrapped return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 53, in select_destinations instance_uuids, return_objects, return_alternates) File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method return getattr(self.instance, __name)(*args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations instance_uuids, return_objects, return_alternates) File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call retry=self.retry) File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send retry=retry) File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send call_monitor_timeout, retry=retry) File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send raise result Environment === detected in Rocky release KVM hypervisor Ceph storage Neutron networks Logs & Configs == in nova.conf: enabled_filters=AggregateInstanceExtraSpecsFilter,RetryFilter,AvailabilityZoneFilter,NUMATopologyFilter,PciPassthroughFilter,RamFilter,ComputeFilter,ImagePropertiesFilter,CoreFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DiskFilter,ComputeCapabilitiesFilter,AggregateRamFilter,SameHostFilter,DifferentHostFilter logs: tbd To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1804502/+subscriptions -- Mailing list:
[Yahoo-eng-team] [Bug 1856923] Re: Virtual machine hot migration failed, no prompt timeout
What is the configuration on your compute nodes? Do you have a timeout set for live migration? Do you have post-copy or auto-converge enabled? What version of libvirt and qemu binaries on those nodes? I'm going to mark this as invalid per a support request until there is more information to help debug this or you've gone through those docs on configuring the nodes for live migration timeout actions. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1856923 Title: Virtual machine hot migration failed, no prompt timeout Status in OpenStack Compute (nova): Invalid Bug description: OpenStack verison Pike. 1.memory_remaining_bytes does not decrease ``` nova server-migration-show 675bcaf1-a9ab-452a-8b0f-be57f5b63cd3 2971 ++--+ | Property | Value | ++--+ | created_at | 2019-12-12T05:38:12.00 | | dest_compute | cmp063 | | dest_host | - | | dest_node | - | | disk_processed_bytes | 0 | | disk_remaining_bytes | 0 | | disk_total_bytes | 0 | | id | 2971 | | memory_processed_bytes | 33290950593 | | memory_remaining_bytes | 20959232 | | memory_total_bytes | 34364792832 | | server_uuid | 675bcaf1-a9ab-452a-8b0f-be57f5b63cd3 | | source_compute | cmp013 | | source_node | - | | status | running | | updated_at | 2019-12-12T05:44:07.00 | ++--+ ``` 2.nova-compute logs No migration process log after 2% memory remaining 3.abort migration failed ``` 2019-12-12 14:31:04,844.844 13398 ERROR nova.virt.libvirt.driver [req-2c8d930b-1a2e-4062-9482-aec0293f675c 78b7f111ca204778858e15d18a3e50a7 d3a063c353814be2957c305396181623 - default default] [instance: 675bcaf1-a9ab-452a-8b0f-be57f5b63cd3] Failed to cancel migration Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainGetJobStats): libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainGetJobStats) 2019-12-12 14:31:04,924.924 13398 ERROR oslo_messaging.rpc.server [req-2c8d930b-1a2e-4062-9482-aec0293f675c 78b7f111ca204778858e15d18a3e50a7 d3a063c353814be2957c305396181623 - default default] Exception during message handling: libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainGetJobStats) ``` To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1856923/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1857013] [NEW] Configure live migrations in nova - timeout section for kvm doesn't mention live_migration_timeout_action
Public bug reported: - [x] This is a doc addition request. In the section on configuring timeouts for live migration for kvm: https://docs.openstack.org/nova/latest/admin/configuring-migrations.html #advanced-configuration-for-kvm-and-qemu It mentions live_migration_completion_timeout but not the live_migration_timeout_action config option which was added in Stein: https://specs.openstack.org/openstack/nova-specs/specs/stein/implemented /live-migration-force-after-timeout.html That allows configuring the compute that if the timeout is reached, to automatically abort (default) or switch to post-copy or auto-converge based on whether or not post-copy is available. --- Release: on 2019-01-16 17:50:10 SHA: 589576ade808eacf69dee4cfcfb4ce48abd04994 Source: https://opendev.org/openstack/nova/src/doc/source/admin/configuring-migrations.rst URL: https://docs.openstack.org/nova/latest/admin/configuring-migrations.html ** Affects: nova Importance: Low Status: Confirmed ** Tags: doc libvirt live-migration low-hanging-fruit ** Tags added: low-hanging-fruit ** Changed in: nova Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1857013 Title: Configure live migrations in nova - timeout section for kvm doesn't mention live_migration_timeout_action Status in OpenStack Compute (nova): Confirmed Bug description: - [x] This is a doc addition request. In the section on configuring timeouts for live migration for kvm: https://docs.openstack.org/nova/latest/admin/configuring- migrations.html#advanced-configuration-for-kvm-and-qemu It mentions live_migration_completion_timeout but not the live_migration_timeout_action config option which was added in Stein: https://specs.openstack.org/openstack/nova- specs/specs/stein/implemented/live-migration-force-after-timeout.html That allows configuring the compute that if the timeout is reached, to automatically abort (default) or switch to post-copy or auto-converge based on whether or not post-copy is available. --- Release: on 2019-01-16 17:50:10 SHA: 589576ade808eacf69dee4cfcfb4ce48abd04994 Source: https://opendev.org/openstack/nova/src/doc/source/admin/configuring-migrations.rst URL: https://docs.openstack.org/nova/latest/admin/configuring-migrations.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1857013/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1856902] [NEW] nova.tests.functional.libvirt.test_vpmem.VPMEMTests.test_create_servers_with_vpmem fails intermittently
Public bug reported:
Seen here:
https://748441c9eeeb89eb36db-
b17eeb704755458252956ad3afd603fe.ssl.cf5.rackcdn.com/697670/3/gate/nova-
tox-functional-py36/54fe49d/testr_results.html.gz
ft1.1:
nova.tests.functional.libvirt.test_vpmem.VPMEMTests.test_create_servers_with_vpmemtesttools.testresult.real._StringException:
pythonlogging:'': {{{
2019-12-18 17:05:18,534 WARNING [placement.db_api] TransactionFactory already
started, not reconfiguring.
2019-12-18 17:05:18,885 INFO [nova.service] Starting conductor node (version
20.1.0)
2019-12-18 17:05:18,917 INFO [nova.service] Starting scheduler node (version
20.1.0)
2019-12-18 17:05:18,981 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET
/v2.1/flavors/detail" status: 200 len: 2605 microversion: 2.81 time: 0.020144
2019-12-18 17:05:18,996 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST
/v2.1/flavors" status: 200 len: 464 microversion: 2.81 time: 0.012445
2019-12-18 17:05:19,018 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST
/v2.1/flavors/99512851/os-extra_specs" status: 200 len: 37 microversion: 2.81
time: 0.018592
2019-12-18 17:05:19,035 INFO [nova.virt.driver] Loading compute driver
'libvirt.LibvirtDriver'
2019-12-18 17:05:19,038 WARNING [os_brick.initiator.connectors.remotefs]
Connection details not present. RemoteFsClient may not initialize properly.
2019-12-18 17:05:19,040 INFO [nova.service] Starting compute node (version
20.1.0)
2019-12-18 17:05:19,058 WARNING [nova.compute.manager] Compute node host1 not
found in the database. If this is the first time this service is starting on
this host, then you can ignore this warning.
2019-12-18 17:05:19,063 INFO [nova.compute.manager] Looking for unclaimed
instances stuck in BUILDING status for nodes managed by this host
2019-12-18 17:05:19,073 WARNING [nova.compute.manager] No compute node record
found for host host1. If this is the first time this service is starting on
this host, then you can ignore this warning.
2019-12-18 17:05:19,082 WARNING [nova.compute.resource_tracker] No compute node
record for host1:host1
2019-12-18 17:05:19,086 INFO [nova.compute.resource_tracker] Compute node
record created for host1:host1 with uuid: 56a25de3-2d91-48a3-9e77-a66f1b5d65b5
2019-12-18 17:05:19,123 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/resource_providers?in_tree=56a25de3-2d91-48a3-9e77-a66f1b5d65b5"
status: 200 len: 26 microversion: 1.14
2019-12-18 17:05:19,131 INFO [placement.requestlog] 127.0.0.1 "POST
/placement/resource_providers" status: 200 len: 828 microversion: 1.20
2019-12-18 17:05:19,132 INFO [nova.scheduler.client.report]
[req-54d75b29-ade3-4f2a-8911-d1147ef449fc] Created resource provider record via
placement API for resource provider with UUID
56a25de3-2d91-48a3-9e77-a66f1b5d65b5 and name host1.
2019-12-18 17:05:19,133 INFO [nova.virt.libvirt.host] kernel doesn't support
AMD SEV
2019-12-18 17:05:19,149 INFO [placement.requestlog] 127.0.0.1 "PUT
/placement/resource_classes/CUSTOM_PMEM_NAMESPACE_SMALL" status: 201 len: 0
microversion: 1.7
2019-12-18 17:05:19,159 INFO [placement.requestlog] 127.0.0.1 "PUT
/placement/resource_classes/CUSTOM_PMEM_NAMESPACE_4GB" status: 201 len: 0
microversion: 1.7
2019-12-18 17:05:19,173 INFO [placement.requestlog] 127.0.0.1 "PUT
/placement/resource_providers/56a25de3-2d91-48a3-9e77-a66f1b5d65b5/inventories"
status: 200 len: 665 microversion: 1.26
2019-12-18 17:05:19,181 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/traits?name=in:COMPUTE_NET_ATTACH_INTERFACE_WITH_TAG,COMPUTE_IMAGE_TYPE_AMI,HW_CPU_X86_VMX,HW_CPU_X86_AESNI,COMPUTE_NET_ATTACH_INTERFACE,COMPUTE_IMAGE_TYPE_AKI,COMPUTE_IMAGE_TYPE_ISO,COMPUTE_VOLUME_ATTACH_WITH_TAG,COMPUTE_IMAGE_TYPE_ARI,COMPUTE_IMAGE_TYPE_RAW,COMPUTE_DEVICE_TAGGING,HW_CPU_HYPERTHREADING,COMPUTE_VOLUME_EXTEND,COMPUTE_IMAGE_TYPE_QCOW2,COMPUTE_TRUSTED_CERTS,COMPUTE_NODE"
status: 200 len: 432 microversion: 1.6
2019-12-18 17:05:19,200 INFO [placement.requestlog] 127.0.0.1 "PUT
/placement/resource_providers/56a25de3-2d91-48a3-9e77-a66f1b5d65b5/traits"
status: 200 len: 467 microversion: 1.6
2019-12-18 17:05:19,224 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET
/v2.1/os-hypervisors?hypervisor_hostname_pattern=host1" status: 200 len: 133
microversion: 2.81 time: 0.019077
2019-12-18 17:05:19,236 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/resource_providers/56a25de3-2d91-48a3-9e77-a66f1b5d65b5/inventories"
status: 200 len: 665 microversion: 1.0
2019-12-18 17:05:19,253 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET
/v2.1/os-hypervisors?hypervisor_hostname_pattern=host1" status: 200 len: 133
microversion: 2.81 time: 0.014420
2019-12-18 17:05:19,255 INFO [nova.tests.functional.libvirt.test_vpmem] booting
on host1
2019-12-18 17:05:19,329 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST
/v2.1/servers" status: 202 len: 414 microversion: 2.81 time: 0.071830
2019-12-18 17:05:19,352 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET
[Yahoo-eng-team] [Bug 1852610] Re: API allows source compute service/node deletion while instances are pending a resize confirm/revert
** Also affects: nova/queens
Importance: Undecided
Status: New
** Changed in: nova/queens
Status: New => Confirmed
** Changed in: nova/stein
Importance: Undecided => Medium
** Changed in: nova/train
Importance: Undecided => Medium
** Changed in: nova/rocky
Importance: Undecided => Medium
** Changed in: nova/queens
Importance: Undecided => Low
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1852610
Title:
API allows source compute service/node deletion while instances are
pending a resize confirm/revert
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) queens series:
Confirmed
Status in OpenStack Compute (nova) rocky series:
In Progress
Status in OpenStack Compute (nova) stein series:
Fix Committed
Status in OpenStack Compute (nova) train series:
Fix Committed
Bug description:
This is split off from bug 1829479 which is about deleting a compute
service which had servers evacuated from it which will orphan resource
providers in placement.
A similar scenario is true where the API will allow deleting a source
compute service which has migration-based allocations for the source
node resource provider and pending instance resizes involving the
source node. A simple scenario is:
1. create a server on host1
2. resize or cold migrate it to a dest host2
3. delete the compute service for host1
At this point the resource provider for host1 is orphaned.
4. try to confirm/revert the resize of the server which will fail
because the compute node for host1 is gone and this results in the
server going to ERROR status
Based on the discussion in this mailing list thread:
http://lists.openstack.org/pipermail/openstack-
discuss/2019-November/010843.html
We should probably have the DELETE /os-services/{service_id} API block
trying to delete a service that has pending migrations.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1852610/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1675791] Re: Instance created by demo user(non-admin), shelved by admin and unshelved by demo user --> ends up in error state
** Changed in: nova/pike Status: Triaged => Won't Fix ** Changed in: nova/queens Status: In Progress => Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1675791 Title: Instance created by demo user(non-admin), shelved by admin and unshelved by demo user --> ends up in error state Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: Won't Fix Status in OpenStack Compute (nova) queens series: Won't Fix Status in OpenStack Compute (nova) rocky series: Fix Committed Bug description: Steps to reproduce === 1) Login as demo user and create an instance. 2) Login as a admin user navigate to admin panel and shelve the instance (as admin user is able shelve any instance). 3) Login as demo user and try to unshelve the instance shelved by admin user. Expected : instance should be unshelved Actual : instance is not shelved but is went to error state. Concerns === There are two conditions here 1.If this scenarios is not valid admin user should not have an option to shelve the instance , this option should be removed . 2.If this is a valid flow , instance should be unsheleved by the demo user. During the shelve process a snap shot will be created in the instance panel and it will be removed automatically when instance is unshelved. But when admin user is trying to shelve instance a snapshot is created under admin projects instead of demo project . This may be the reason for unshelve failure Admin user is able to unsheleve an instance shelved by demo user as he is seeing both snapshots. Environment === Reproduced it with pure stable/Newton in devstack environment Also reproduced it with Liberty. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1675791/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1788527] Re: Redundant instance group lookup during scheduling of move operations
** Changed in: nova/rocky Status: In Progress => Confirmed ** Changed in: nova/rocky Status: Confirmed => Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1788527 Title: Redundant instance group lookup during scheduling of move operations Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) rocky series: Won't Fix Status in OpenStack Compute (nova) stein series: Fix Committed Bug description: This change: https://github.com/openstack/nova/commit/459ca56de2366aea53efc9ad3295fdf4ddcd452c Added code to the setup_instance_group flow to get the instance group fresh so we had the latest hosts for members of the group. Then change: https://github.com/openstack/nova/commit/94fd36f0582c5dbcf2b9886da7c7bf986d3ad5d1 #diff-cbbdc4d7c140314a7e0b2d97ebcd1f9c Was added to not persist group hosts/members in the RequestSpec since they could be stale after the initial server create. This means when we move a server (evacuate, resize, unshelve, live migrate), we get the request spec with the group plus the current hosts/members of the group. So if the request spec has the group hosts set by the time it gets to setup_instance_group, the call in _get_group_details to get the group fresh is redundant. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1788527/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840068] Re: (lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len() - python3
** Also affects: nova/pike Importance: Undecided Status: New ** Changed in: nova/pike Status: New => In Progress ** Changed in: nova/pike Assignee: (unassigned) => sean mooney (sean-k-mooney) ** Changed in: nova/pike Importance: Undecided => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1840068 Title: (lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len() - python3 Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: In Progress Status in OpenStack Compute (nova) queens series: Fix Committed Status in OpenStack Compute (nova) rocky series: Fix Committed Status in OpenStack Compute (nova) stein series: Fix Committed Bug description: Seen in the nova-lxc CI job here: https://logs.opendev.org/24/676024/2/experimental/nova- lxc/f9a892c/controller/logs/screen-n-cpu.txt.gz#_Aug_12_23_31_05_043911 Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [None req-55d6dd1b-96ca-4afe-9a0c-cec902d3bd87 tempest-ServerAddressesTestJSON-1311986476 tempest-ServerAddressesTestJSON-1311986476] [instance: 842a9704-3700-42ef-adb3-b038ca525127] Instance failed to spawn: TypeError: object of type 'filter' has no len() Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] Traceback (most recent call last): Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/compute/manager.py", line 2495, in _build_resources Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] yield resources Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/compute/manager.py", line 2256, in _build_and_run_instance Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] block_device_info=block_device_info) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3231, in spawn Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] destroy_disks_on_failure=True) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5823, in _create_domain_and_network Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] destroy_disks_on_failure) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] self.force_reraise() Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] six.reraise(self.type_, self.value, self.tb) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] raise value Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5789, in _create_domain_and_network Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR
[Yahoo-eng-team] [Bug 1724172] Re: Allocation of an evacuated instance is not cleaned on the source host if instance is not defined on the hypervisor
** Also affects: nova/pike Importance: Undecided Status: New ** Changed in: nova/pike Status: New => In Progress ** Changed in: nova/pike Importance: Undecided => Low ** Changed in: nova/pike Assignee: (unassigned) => Balazs Gibizer (balazs-gibizer) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1724172 Title: Allocation of an evacuated instance is not cleaned on the source host if instance is not defined on the hypervisor Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: In Progress Status in OpenStack Compute (nova) queens series: Fix Committed Status in OpenStack Compute (nova) rocky series: Fix Committed Status in OpenStack Compute (nova) stein series: Fix Committed Bug description: Nova does not clean up the allocation of an evacuated instance from the recovered source compute host if the instance is not any more defined on the hypervisor. To reproduce: * Boot an instance * Kill the compute host the instance is booted on * Evacuate the instance * Recover the original compute host in a way that clears the instance definition from the hypervisor (e.g. redeploy the compute host). * Check the allocations of the instance in placement API. The allocation against the source compute host is not cleaned up. The compute manager is supposed to clean up evacuated instances during the compute manager init_host method by calling _destroy_evacuated_instances. However that function only iterates on instances known by the hypervisor [1]. [1] https://github.com/openstack/nova/blob/5e4c98a58f1afeaa903829f5e3f28cd6dc30bf4b/nova/compute/manager.py#L654 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1724172/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1856296] Re: upgrade to Train might fail due to mariadb row format
I don't think there is really anything for nova to do with this, it's dependent on how the database is configured correct? ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1856296 Title: upgrade to Train might fail due to mariadb row format Status in kolla-ansible: Triaged Status in OpenStack Compute (nova): Invalid Bug description: In kolla-ansible CI we started getting on Ubuntu for Stein->Train: Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs for "ALTER TABLE instances ADD hidden BOOL" in nova upgrade This is due to the following: ... If the table is using either the REDUNDANT or the COMPACT row format, then one potential solution to this problem is to convert the table to use the DYNAMIC row format instead. If your tables were originally created on an older version of MariaDB or MySQL, then your table may be using one of InnoDB's older row formats: In MariaDB 10.1 and before, and in MySQL 5.6 and before, the COMPACT row format was the default row format. ... from https://mariadb.com/kb/en/library/troubleshooting-row-size-too-large-errors-with-innodb/#solving-the-problem Indeed, the Stein images for Ubuntu include MariaDB 10.1 which we had to upgrade in Train due to other incompatibilities (with neutron). It is also worth noting that old CentOS deployments may also have this limited row format. To manage notifications about this bug go to: https://bugs.launchpad.net/kolla-ansible/+bug/1856296/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1856312] Re: RuntimeError during calling log_opts_values
** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1856312 Title: RuntimeError during calling log_opts_values Status in OpenStack Compute (nova): Invalid Status in oslo.config: In Progress Bug description: During starting up nova-compute service, we are hit by the following error message + sed -i s/HOST_IP// /tmp/logging-nova-compute.conf + exec nova-compute --config-file /etc/nova/nova.conf --config-file /tmp/pod-shared/nova-console.conf --config-file /tmp/pod-shared/nova-libvirt.conf --config-file /tmp/pod-shared/nova-hypervisor.conf --log-config-append /tmp/logging-nova-compute.conf 2019-12-13 06:53:09.556 29036 WARNING oslo_config.cfg [-] Deprecated: Option "use_neutron" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-12-13 06:53:12.000 29036 INFO nova.compute.rpcapi [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Automatically selected compute RPC version 5.0 from minimum service version 35 2019-12-13 06:53:12.000 29036 INFO nova.compute.rpcapi [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Automatically selected compute RPC version 5.0 from minimum service version 35 2019-12-13 06:53:12.029 29036 INFO nova.virt.driver [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Loading compute driver 'libvirt.LibvirtDriver' 2019-12-13 06:53:12.029 29036 INFO nova.virt.driver [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Loading compute driver 'libvirt.LibvirtDriver' 2019-12-13 06:53:22.064 29036 WARNING oslo_config.cfg [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Deprecated: Option "firewall_driver" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-12-13 06:53:22.192 29036 WARNING os_brick.initiator.connectors.remotefs [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Connection details not present. RemoteFsClient may not initialize properly. 2019-12-13 06:53:22.409 29036 WARNING oslo_config.cfg [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Deprecated: Option "linuxnet_interface_driver" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-12-13 06:53:22.414 29036 WARNING oslo_config.cfg [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Deprecated: Option "metadata_port" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-12-13 06:53:22.440 29036 INFO nova.service [-] Starting compute node (version 18.0.0) 2019-12-13 06:53:22.570 29036 WARNING oslo_config.cfg [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Deprecated: Option "api_endpoint" from group "ironic" is deprecated for removal (Endpoint lookup uses the service catalog via common keystoneauth1 Adapter configuration options. In the current release, api_endpoint will override this behavior, but will be ignored and/or removed in a future release. To achieve the same result, use the endpoint_override option instead.). Its value may be silently ignored in the future. 2019-12-13 06:53:22.440 29036 INFO nova.service [-] Starting compute node (version 18.0.0) 2019-12-13 06:53:22.594 29036 WARNING oslo_config.cfg [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Deprecated: Option "api_endpoint" from group "ironic" is deprecated. Use option "endpoint-override" from group "ironic". 2019-12-13 06:53:22.911 29036 CRITICAL nova [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Unhandled error: RuntimeError: dictionary changed size during iteration 2019-12-13 06:53:22.911 29036 ERROR nova Traceback (most recent call last): 2019-12-13 06:53:22.911 29036 ERROR nova File "/var/lib/openstack/bin/nova-compute", line 8, in 2019-12-13 06:53:22.911 29036 ERROR nova sys.exit(main()) 2019-12-13 06:53:22.911 29036 ERROR nova File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/inspur/cmd/compute.py", line 71, in main 2019-12-13 06:53:22.911 29036 ERROR nova service.wait() 2019-12-13 06:53:22.911 29036 ERROR nova File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/service.py", line 460, in wait 2019-12-13 06:53:22.911 29036 ERROR nova _launcher.wait() 2019-12-13 06:53:22.911 29036 ERROR nova File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_service/service.py", line 392, in wait 2019-12-13 06:53:22.911 29036 ERROR nova status, signo = self._wait_for_exit_or_signal() 2019-12-13
[Yahoo-eng-team] [Bug 1856312] Re: RuntimeError during calling log_opts_values
Does this happen every time and is 100% reproducible or is it intermittent somehow? ** Also affects: oslo.config Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1856312 Title: RuntimeError during calling log_opts_values Status in OpenStack Compute (nova): New Status in oslo.config: New Bug description: During starting up nova-compute service, we are hit by the following error message + sed -i s/HOST_IP// /tmp/logging-nova-compute.conf + exec nova-compute --config-file /etc/nova/nova.conf --config-file /tmp/pod-shared/nova-console.conf --config-file /tmp/pod-shared/nova-libvirt.conf --config-file /tmp/pod-shared/nova-hypervisor.conf --log-config-append /tmp/logging-nova-compute.conf 2019-12-13 06:53:09.556 29036 WARNING oslo_config.cfg [-] Deprecated: Option "use_neutron" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-12-13 06:53:12.000 29036 INFO nova.compute.rpcapi [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Automatically selected compute RPC version 5.0 from minimum service version 35 2019-12-13 06:53:12.000 29036 INFO nova.compute.rpcapi [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Automatically selected compute RPC version 5.0 from minimum service version 35 2019-12-13 06:53:12.029 29036 INFO nova.virt.driver [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Loading compute driver 'libvirt.LibvirtDriver' 2019-12-13 06:53:12.029 29036 INFO nova.virt.driver [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Loading compute driver 'libvirt.LibvirtDriver' 2019-12-13 06:53:22.064 29036 WARNING oslo_config.cfg [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Deprecated: Option "firewall_driver" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-12-13 06:53:22.192 29036 WARNING os_brick.initiator.connectors.remotefs [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Connection details not present. RemoteFsClient may not initialize properly. 2019-12-13 06:53:22.409 29036 WARNING oslo_config.cfg [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Deprecated: Option "linuxnet_interface_driver" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-12-13 06:53:22.414 29036 WARNING oslo_config.cfg [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Deprecated: Option "metadata_port" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-12-13 06:53:22.440 29036 INFO nova.service [-] Starting compute node (version 18.0.0) 2019-12-13 06:53:22.570 29036 WARNING oslo_config.cfg [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Deprecated: Option "api_endpoint" from group "ironic" is deprecated for removal (Endpoint lookup uses the service catalog via common keystoneauth1 Adapter configuration options. In the current release, api_endpoint will override this behavior, but will be ignored and/or removed in a future release. To achieve the same result, use the endpoint_override option instead.). Its value may be silently ignored in the future. 2019-12-13 06:53:22.440 29036 INFO nova.service [-] Starting compute node (version 18.0.0) 2019-12-13 06:53:22.594 29036 WARNING oslo_config.cfg [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Deprecated: Option "api_endpoint" from group "ironic" is deprecated. Use option "endpoint-override" from group "ironic". 2019-12-13 06:53:22.911 29036 CRITICAL nova [req-eec76cc3-35a9-4d1d-bb91-4c484f6ef855 - - - - -] Unhandled error: RuntimeError: dictionary changed size during iteration 2019-12-13 06:53:22.911 29036 ERROR nova Traceback (most recent call last): 2019-12-13 06:53:22.911 29036 ERROR nova File "/var/lib/openstack/bin/nova-compute", line 8, in 2019-12-13 06:53:22.911 29036 ERROR nova sys.exit(main()) 2019-12-13 06:53:22.911 29036 ERROR nova File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/inspur/cmd/compute.py", line 71, in main 2019-12-13 06:53:22.911 29036 ERROR nova service.wait() 2019-12-13 06:53:22.911 29036 ERROR nova File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/service.py", line 460, in wait 2019-12-13 06:53:22.911 29036 ERROR nova _launcher.wait() 2019-12-13 06:53:22.911 29036 ERROR nova File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_service/service.py", line 392, in wait 2019-12-13
[Yahoo-eng-team] [Bug 1856311] Re: server_external_events response status is always 'completed' instead of event status which requested as failed
Hmm this should pull the status from the request and put it on the
response:
https://github.com/openstack/nova/blob/e6f742544432d6066f1fba4666580919eb7859bd/nova/api/openstack/compute/server_external_events.py#L96
and then changed to completed here because the instance.host is set so
we're going to process the event:
https://github.com/openstack/nova/blob/e6f742544432d6066f1fba4666580919eb7859bd/nova/api/openstack/compute/server_external_events.py#L133
I think this is working as designed.
** Changed in: nova
Status: New => Opinion
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1856311
Title:
server_external_events response status is always 'completed' instead
of event status which requested as failed
Status in OpenStack Compute (nova):
Opinion
Bug description:
Description
===
For the server external event rest api response is always completed instead
of what the user requested to updated as failed/in-progress .
here the changes required :
https://github.com/openstack/nova/blob/aa096fd18352fb9da94069ec2cab478eed5c6cca/nova/api/openstack/compute/server_external_events.py#L133
Steps to reproduce
==
1. POST a rest api request with the status as failed .
2. API response event status will be always completed .
Expected result
===
Post request :
openstack/compute/v2.1/os-server-external-events
{
"events": [
{
"name": "network-vif-plugged",
"tag": "foo",
"server_uuid": "b20f436b-b9b6-4a8d-a1f7-411ed42ffe62",
"status": "failed"
}
]
}
Response :
{
"events": [
{
"status": "failed",
"tag": "foo",
"name": "network-vif-plugged",
"server_uuid": "b20f436b-b9b6-4a8d-a1f7-411ed42ffe62",
"code": 200
}
]
}
Actual result
=
Post request :
openstack/compute/v2.1/os-server-external-events
{
"events": [
{
"name": "network-vif-plugged",
"tag": "foo",
"server_uuid": "b20f436b-b9b6-4a8d-a1f7-411ed42ffe62",
"status": "failed"
}
]
}
Response :
{
"events": [
{
"status": "completed",
"tag": "foo",
"name": "network-vif-plugged",
"server_uuid": "b20f436b-b9b6-4a8d-a1f7-411ed42ffe62",
"code": 200
}
]
}
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1856311/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1695187] Re: bdm save failure leaves inconsistent data during attaching volume
This is really old and needs to be re-triaged/re-reported if it's still
a valid issue on master.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1695187
Title:
bdm save failure leaves inconsistent data during attaching volume
Status in OpenStack Compute (nova):
Invalid
Bug description:
The bug is found in Kilo environment. It exists in master.
Before saving the bdm, the volume has been attached at the hypervisor level.
When save fails, you will see the VM has virtual disk (virsh domblklist)
but can't see it when you do nova show or cinder show.
Even worse, in my IP-SAN environment, the configuration isn't cleaned at the
array,
which might make data inconsistent.
nova/virt/block_device.py
@update_db
def attach(self, context, instance, volume_api, virt_driver,
do_driver_attach=False, **kwargs):
.
if volume['attach_status'] == "detached":
# NOTE(mriedem): save our current state so connection_info is in
# the database before the volume status goes to 'in-use' because
# after that we can detach and connection_info is required for
# detach.
self.save() < Errors here !!!
try:
volume_api.attach(context, volume_id, instance.uuid,
self['mount_device'], mode=mode)
except Exception:
with excutils.save_and_reraise_exception():
if do_driver_attach:
try:
virt_driver.detach_volume(connection_info,
instance,
self['mount_device'],
encryption=encryption)
except Exception:
LOG.warning(_LW("Driver failed to detach volume "
"%(volume_id)s at %(mount_point)s."),
{'volume_id': volume_id,
'mount_point': self['mount_device']},
exc_info=True, instance=instance)
volume_api.terminate_connection(context, volume_id,
connector)
The trace:
--
2017-05-24 17:24:23.272 3125 ERROR nova.compute.manager
[req-6040188f-4338-47a1-855d-4ecc0eb71203 62f52135115f4898bd0d82c1f0cd632b
6c149dcd3cf64171b8dd972dd03bbac0 - - -] [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] Failed to attach
2421acfd-0f94-4aca-81d0-747bf803aed7 at /dev/vdb
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] Traceback (most recent call last):
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5226, in
_attach_volume
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] do_check_attach=False,
do_driver_attach=True)
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 53, in
wrapped
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] obj.save()
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 321, in save
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] super(DriverVolumeBlockDevice,
self).save()
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 143, in save
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] self._bdm_obj.save()
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] File
"/usr/lib/python2.7/site-packages/nova/objects/base.py", line 192, in wrapper
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] self._context, self, fn.__name__,
args, kwargs)
2017-05-24 17:24:23.272 3125 TRACE nova.compute.manager [instance:
63b7baae-599c-41b3-bbee-f59bbc239e57] File
"/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 340, in
object_action
[Yahoo-eng-team] [Bug 1769025] Re: OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object error is thrown for ironic deploy in Pike
Is this still an issue? I'm inclined to mark this as invalid unless it can be recreated on a newer release. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1769025 Title: OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object error is thrown for ironic deploy in Pike Status in OpenStack Compute (nova): Invalid Bug description: On the Pike setup, nova boot for ironic fails consistently due to following error: 2018-04-13 07:56:02.846 DEBUG oslo_concurrency.processutils [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] CMD "genisoimage -o /tmp/tmppY5Ikz -ldots -allow-low ercase -allow-multidot -l -publisher OpenStack Nova 16.1.1 -quiet -J -r -V config-2 /tmp/tmpxFZUMX" returned: 0 in 0.019s from (pid=47123) execute /usr/local/lib/pytho n2.7/dist-packages/oslo_concurrency/processutils.py:404 2018-04-13 07:56:02.863 INFO nova.virt.ironic.driver [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Config drive for instance 3b1195b4-3106-4c5c-87f4-69d203d08 884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96 created. 2018-04-13 07:56:03.656 ERROR oslo.service.loopingcall [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Fixed interval looping call 'nova.virt.ironic.driver.Iron icDriver._wait_for_active' failed: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object Traceback (most recent call last): File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper return fn(self, *args, **kwargs) File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh elif self[field] != current[field]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__ return getattr(self, name) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter self.obj_load_attr(name) File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr objtype=self.obj_name()) OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last): 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall result = func(*self.args, **self.kw) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall instance.refresh() 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall ctxt, self, fn.__name__, args, kwargs) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objmethod=objmethod, args=args, kwargs=kwargs) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=self.retry) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall timeout=timeout, retry=retry) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=retry) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall raise result 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last): 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return
[Yahoo-eng-team] [Bug 1856241] [NEW] Compute API in nova - servers_links should link to paging doc
Public bug reported: - [x] This doc is inaccurate in this way: The servers_links parameter description says: """ Links to the next server. It is available when the number of servers exceeds limit parameter or [api]/max_limit in the configuration file. See API Guide / Links and References for more info. """ The link to the API guide goes here: https://docs.openstack.org/api-guide/compute/links_and_references.html But that doesn't talk about paging or servers_links. The API reference description should link here for talking about paging: https://docs.openstack.org/api-guide/compute/paginated_collections.html --- Release: on 2019-10-08 09:22:51 SHA: b6107c8b97bc6a662237873098179841b766ab19 Source: https://opendev.org/openstack/nova/src/api-ref/source/index.rst URL: https://docs.openstack.org/api-ref/compute/ ** Affects: nova Importance: Low Status: Triaged ** Tags: api-ref doc low-hanging-fruit ** Changed in: nova Status: New => Triaged ** Changed in: nova Importance: Undecided => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1856241 Title: Compute API in nova - servers_links should link to paging doc Status in OpenStack Compute (nova): Triaged Bug description: - [x] This doc is inaccurate in this way: The servers_links parameter description says: """ Links to the next server. It is available when the number of servers exceeds limit parameter or [api]/max_limit in the configuration file. See API Guide / Links and References for more info. """ The link to the API guide goes here: https://docs.openstack.org/api-guide/compute/links_and_references.html But that doesn't talk about paging or servers_links. The API reference description should link here for talking about paging: https://docs.openstack.org/api- guide/compute/paginated_collections.html --- Release: on 2019-10-08 09:22:51 SHA: b6107c8b97bc6a662237873098179841b766ab19 Source: https://opendev.org/openstack/nova/src/api-ref/source/index.rst URL: https://docs.openstack.org/api-ref/compute/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1856241/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1856240] [NEW] wait_for_versioned_notifications gives unhelpful error message "ValueError: Not a text type application/octet-stream" on timeout
Public bug reported:
While working on a functional test that relies on the
fake_notifier.wait_for_versioned_notifications method the notification
never comes and the waiter times out but dumps an unhelpful ValueError:
{0}
nova.tests.functional.wsgi.test_servers.ColdMigrationDisallowSameHost.test_cold_migrate_same_host_disabled
[] ... inprogress
Captured traceback:
~~~
b'Traceback (most recent call last):'
b' File
"/home/osboxes/git/nova/nova/tests/functional/wsgi/test_servers.py", line 423,
in test_cold_migrate_same_host_disabled'
b'self._wait_for_migrate_no_valid_host()'
b' File
"/home/osboxes/git/nova/nova/tests/functional/wsgi/test_servers.py", line 408,
in _wait_for_migrate_no_valid_host'
b"'compute_task.migrate_server.error')[0]"
b' File "/home/osboxes/git/nova/nova/tests/unit/fake_notifier.py", line
153, in wait_for_versioned_notifications'
b'return VERSIONED_SUBS[event_type].wait_n(n_events, event_type,
timeout)'
b' File "/home/osboxes/git/nova/nova/tests/unit/fake_notifier.py", line
61, in wait_n'
b"'notifications': notifications,"
b''
Not a text type application/octet-stream
Traceback (most recent call last):
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/cliff/app.py",
line 401, in run_subcommand
result = cmd.run(parsed_args)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/cliff/command.py",
line 185, in run
return_code = self.take_action(parsed_args) or 0
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/stestr/commands/run.py",
line 235, in take_action
all_attachments=all_attachments)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/stestr/commands/run.py",
line 484, in run_command
all_attachments=all_attachments)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/stestr/commands/run.py",
line 550, in _run_tests
return run_tests()
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/stestr/commands/run.py",
line 547, in run_tests
all_attachments=all_attachments)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/stestr/commands/load.py",
line 234, in load
all_attachments)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/stestr/commands/load.py",
line 267, in _load_case
case.run(result)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/testsuite.py",
line 171, in run
result.status(**event_dict)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/testresult/real.py",
line 468, in status
_strict_map(methodcaller('status', *args, **kwargs), self.targets)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/testresult/real.py",
line 443, in _strict_map
return list(map(function, *sequences))
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/testresult/real.py",
line 570, in status
target.status(**kwargs)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/testresult/real.py",
line 468, in status
_strict_map(methodcaller('status', *args, **kwargs), self.targets)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/testresult/real.py",
line 443, in _strict_map
return list(map(function, *sequences))
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/testresult/real.py",
line 909, in status
self._hook.status(*args, **kwargs)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/testresult/real.py",
line 826, in status
self.on_test(self._inprogress.pop(key))
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/testresult/real.py",
line 901, in _handle_test
self.on_test(test_record.to_dict())
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/stestr/subunit_trace.py",
line 193, in show_outcome
print_attachments(stream, test, all_channels=True)
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/stestr/subunit_trace.py",
line 120, in print_attachments
if (all_channels or name in channels) and detail.as_text():
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/content.py",
line 92, in as_text
return _u('').join(self.iter_text())
File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/content.py",
line 108, in iter_text
raise ValueError("Not a text type %r" % self.content_type)
ValueError: Not a text type application/octet-stream
There is a problem formating the AssertionError which is trying to
[Yahoo-eng-team] [Bug 1827489] Re: Wrong IPV6 address provided by openstack server create
Marking this as invalid for nova since it doesn't appear there is anything for nova to do with this: (12:55:13 PM) mriedem: haleyb: do you think there is anything for nova to do with this? https://bugs.launchpad.net/nova/+bug/1827489 (12:55:15 PM) openstack: Launchpad bug 1827489 in neutron "Wrong IPV6 address provided by openstack server create" [Low,In progress] - Assigned to Brian Haley (brian-haley) (1:00:35 PM) haleyb: mriedem: no, it's really a config issue in the guest (1:01:02 PM) sean-k-mooney: mriedem: this seam ver config specific. when you enable ipv6 in neutron you have several option includign stateful dhcp6 ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1827489 Title: Wrong IPV6 address provided by openstack server create Status in neutron: In Progress Status in OpenStack Compute (nova): Invalid Bug description: IPV6 address of an interface doesn't have to be derived from its MAC address. The newer kernels have addr_gen_mode option which controls the behavior of IPV6 calculation, see https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt I've encountered the problem when I booted up an image (RHEL8 in my case) which had the addr_gen_mode option set to 1 (means that IPV6 address is randomized) by default. OpenStack (I had Rocky deployment) didn't recognize this and 'openstack server create' returned wrong address which lead to tempest failures because thanks to the 'openstack server create' output the tests expected different addresses on the interfaces. Steps to reproduce: $ openstack server create --image --flavor --network --network --key-name instance_name +-++ | Field | Value | +-++ | accessIPv4 | | | accessIPv6 | | | addresses | tempest-network-smoke--884367252=10.100.0.5; tempest-network-smoke--18828977=2003::f816:3eff:febb:7456 | Then ssh to the instance and hit 'ip a' command: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1450 qdisc fq_codel state UP group default qlen 1000 link/ether fa:16:3e:48:e8:b5 brd ff:ff:ff:ff:ff:ff inet 10.100.0.3/28 brd 10.100.0.15 scope global dynamic noprefixroute eth0 valid_lft 86363sec preferred_lft 86363sec inet6 fe80::f816:3eff:fe48:e8b5/64 scope link valid_lft forever preferred_lft forever 3: eth1: mtu 1450 qdisc fq_codel state UP group default qlen 1000 link/ether fa:16:3e:bb:74:56 brd ff:ff:ff:ff:ff:ff inet6 2003::b47f:f400:ecca:2a55/64 scope global dynamic noprefixroute valid_lft 86385sec preferred_lft 14385sec inet6 fe80::7615:8d57:775d:fae/64 scope link noprefixroute valid_lft forever preferred_lft forever Notice that eth1 interface has an ipv6 address which seems not to be derived from its mac address. Also notice that the output of 'openstack server create' returned wrong address, a different one than it's actually set for eth1. It expected that the ipv6 address would be derived from the mac address but it wasn't. 'openstack server create' should be able to detect the option in the image and behave accordingly. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1827489/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1842909] Re: The vm is not assigned security group when created with port-id option
Agree that this is extremely latent behavior and I'm pretty sure we have
existing/closed/invalidated bugs for orchestrating this type of
behavior. As James noted from the docs, pre-existing ports get the
security groups applied to them in neutron. I'm going to close this as
Won't Fix and a Wishlist bug. If this is something that is really
desired it should be reported as a blueprint/spec in my opinion so there
can be a discussion in gerrit about the merits of this request for
orchestration, and note:
https://docs.openstack.org/nova/latest/contributor/project-scope.html
#no-more-orchestration
There are sometimes exceptions to that policy, like adding the ability
to boot-from-volume and provide a volume type for nova to use when it
creates volumes on the user's behalf, but those are exceptions due to
high demand from multiple users/vendors.
** Changed in: nova
Status: New => Won't Fix
** Changed in: nova
Importance: Undecided => Wishlist
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1842909
Title:
The vm is not assigned security group when created with port-id option
Status in OpenStack Compute (nova):
Won't Fix
Bug description:
>> I have created a neutron network and a subnet
>> I have created a port on the network
>> I have created a vm with the port id option with --security-group option
provided
>>The CLI used.
nova --insecure boot --image cirros --flavor m1.tiny --nic
port-id=f6c035a3-fd93-4734-8210-6b64c4d0e66c vm-y --security-group sg1
>>But when i check the port of the vm, the security group sg1 is not
applied.
root@prome-mdt-dhcp412:~# neutron port-show
f6c035a3-fd93-4734-8210-6b64c4d0e66c
+--+--+
| Field| Value
|
+--+--+
| admin_state_up | True
|
| allowed_address_pairs|
|
| binding:host_id | compute-c99bffcb-c8
|
| binding:profile | {}
|
| binding:vif_details | {"ovs_hybrid_plug": false,
"nsx-logical-switch-id": "c7474c18-611f-421d-bb3f-176aca21841e", "port_filter":
true} |
| binding:vif_type | ovs
|
| binding:vnic_type| normal
|
| created_at | 2019-09-05T07:22:34Z
|
| description |
|
| device_id| 3ee5ea9b-a0ea-4e51-a3cb-6c2e54382fee
|
| device_owner | compute:nova
|
| extra_dhcp_opts |
|
| fixed_ips| {"subnet_id":
"ed327c19-c928-4de3-adea-6be9c3d9f80e", "ip_address": "13.0.0.16"}
|
| id | f6c035a3-fd93-4734-8210-6b64c4d0e66c
|
| mac_address | fa:16:3e:c8:d8:f1
|
| name | port-y
|
| network_id | 274a0665-08dc-4a27-9be0-636718576757
|
| port_security_enabled| True
|
| project_id | 0e551202bb7644c68b89dda3db23d487
[Yahoo-eng-team] [Bug 1851336] Re: cirros aarch64 img got vnc problem: Guest disabled display
> I think this not a bug for Nova, we need report the issue in cirros project. so we can close this one Done. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1851336 Title: cirros aarch64 img got vnc problem: Guest disabled display Status in OpenStack Compute (nova): Invalid Bug description: I used OpenStack Newton on aarch64 platform. After backporting patches support for aarch64 from OpenStack upstream, I created instance with cirros-0.4.0-aarch64-disk.img, the instance could be active, but guest system didn't boot up. I tried to remove acpi from libvirt config, the guest system could boot up, but I got a new problem: vnc console on dashboard was failed, the screen printed "Guest disabled display". libvirt: 3.2.0 qemu: 2.9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1851336/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1856147] Re: Unexpected API Error when create server
Looks like misconfiguration where nova is unable to communicate with
another service. Check your install and config steps.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1856147
Title:
Unexpected API Error when create server
Status in OpenStack Compute (nova):
Invalid
Bug description:
Description
===
when i tried to create server user openstack client, error reported.
Steps to reproduce
==
* create image
* create flavor
* create network and subnet
* create server
command:
openstack --debug server create --image 75067e32-3812-4c4c-
a8c2-0a1338790a93 --flavor 7cbe3e6a-85d9-4a31-a78c-6fba32a94e45
--network 54e62c88-1345-4f31-aed7-5f602d690f7f xy-test
Expected result
===
create a server
Actual result
=
ClientException: Unexpected API Error. Please report this at
http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
(HTTP 500) (Request-ID:
req-2a1cff4a-b7ee-4a3b-b02f-76adfe96dd80)
Environment
===
1. version:
openstack-rocky
openstack-nova-scheduler-18.2.3-1.el7.noarch
openstack-nova-api-18.2.3-1.el7.noarch
openstack-nova-console-18.2.3-1.el7.noarch
openstack-nova-conductor-18.2.3-1.el7.noarch
python2-novaclient-11.0.1-1.el7.noarch
python-nova-18.2.3-1.el7.noarch
openstack-nova-novncproxy-18.2.3-1.el7.noarch
openstack-nova-common-18.2.3-1.el7.noarch
openstack-nova-compute-18.2.3-1.el7.noarch
openstack-nova-placement-api-18.2.3-1.el7.noarch
2. Which hypervisor did you use?
libvirt + KVM
2. Which storage type did you use?
LVM
lvm2-2.02.185-2.el7_7.2.x86_64
llvm-private-7.0.1-1.el7.x86_64
lvm2-libs-2.02.185-2.el7_7.2.x86_64
3. Which networking type did you use?
neutron
Logs & Configs
==
boot_args: [u'xy-test', , ]
boot_kwargs: {'files': {}, 'userdata': None, 'key_name': None,
'availability_zone': None, 'nics': [{'port-id': '', 'net-id':
u'54e62c88-1345-4f31-aed7-5f602d690f7f', 'v4-fixed-ip': '', 'v6-fixed-ip':
''}], 'max_count': 1, 'meta': None, 'block_device_mapping_v2': [], 'min_count':
1, 'scheduler_hints': {}, 'reservation_id': None, 'security_groups': [],
'config_drive': None}
REQ: curl -g -i -X POST http://controller:8774/v2.1/servers -H "Accept:
application/json" -H "Content-Type: application/json" -H "User-Agent:
python-novaclient" -H "X-Auth-Token:
{SHA1}40ac1edceb6b28070c6a1fd58355a35dffe3e41a" -d '{"server": {"name":
"xy-test", "imageRef": "75067e32-3812-4c4c-a8c2-0a1338790a93", "flavorRef":
"7cbe3e6a-85d9-4a31-a78c-6fba32a94e45", "max_count": 1, "min_count": 1,
"networks": [{"uuid": "54e62c88-1345-4f31-aed7-5f602d690f7f"}]}}'
http://controller:8774 "POST /v2.1/servers HTTP/1.1" 500 216
RESP: [500] Connection: keep-alive Content-Length: 216 Content-Type:
application/json; charset=UTF-8 Date: Thu, 12 Dec 2019 08:13:12 GMT
Openstack-Api-Version: compute 2.1 Vary: OpenStack-API-Version,
X-OpenStack-Nova-API-Version X-Compute-Request-Id:
req-10cb460c-d4eb-43ed-b327-d7af88151d70 X-Openstack-Nova-Api-Version: 2.1
X-Openstack-Request-Id: req-10cb460c-d4eb-43ed-b327-d7af88151d70
RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report
this at http://bugs.launchpad.net/nova/ and attach the Nova API log if
possible.\n", "code": 500}}
POST call to compute for http://controller:8774/v2.1/servers used request id
req-10cb460c-d4eb-43ed-b327-d7af88151d70
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/
and attach the Nova API log if possible.
(HTTP 500) (Request-ID:
req-10cb460c-d4eb-43ed-b327-d7af88151d70)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in
run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line
41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/display.py", line 116, in run
column_names, data = self.take_action(parsed_args)
File
"/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line
917, in take_action
server = compute_client.servers.create(*boot_args, **boot_kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line
1327, in create
return self._boot(response_key, *boot_args, **boot_kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 776,
in _boot
return_raw=return_raw, **kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 366, in
_create
resp, body = self.api.client.post(url, body=body)
File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334,
in post
return self.request(url,
[Yahoo-eng-team] [Bug 1855927] [NEW] _poll_unconfirmed_resizes may not retry later if confirm_resize fails in API
Public bug reported: This is based on code inspection but let's say I have configured my computes to set resize_confirm_window=3600 to automatically confirm a resized server after 1 hour. Within that hour, let's say the source compute service is down. The periodic task gets the unconfirmed migrations with status='finished' which have been updated some time older than the given configurable window: https://github.com/openstack/nova/blob/5a3ef39539ca112ae0552aef5cbd536338db61b7/nova/compute/manager.py#L8793 https://github.com/openstack/nova/blob/5a3ef39539ca112ae0552aef5cbd536338db61b7/nova/db/sqlalchemy/api.py#L4342 The periodic task then calls the compute API code to confirm the resize: https://github.com/openstack/nova/blob/c295e395d/nova/compute/manager.py#L7160 which changes the migration status to 'confirming': https://github.com/openstack/nova/blob/5a3ef39539ca112ae0552aef5cbd536338db61b7/nova/compute/api.py#L3684 And casts off to the source compute: https://github.com/openstack/nova/blob/5a3ef39539ca112ae0552aef5cbd536338db61b7/nova/compute/rpcapi.py#L600 Now if the source compute is down and that fails, the compute manager task code will handle it and say it will retry later: https://github.com/openstack/nova/blob/c295e395d/nova/compute/manager.py#L7163 However, because the migration status was changed from 'finished' to 'confirming' the task will not retry because it won't find the migration given the DB query. And trying to confirm the resize via the API will fail as well because we'll get MigrationNotFoundByStatus since the migration status is no longer 'finished': https://github.com/openstack/nova/blob/5a3ef39539ca112ae0552aef5cbd536338db61b7/nova/compute/api.py#L3681 The compute manager code should probably mark the migration status as 'finished' again if it's really going to try later, or mark the migration status as 'error'. Note that the confirm_resize method in the compute manager doesn't mark the migration status as 'error' if something fails there either: https://github.com/openstack/nova/blob/c295e395d/nova/compute/manager.py#L3807 ** Affects: nova Importance: Low Status: New ** Tags: error-handling migrate resize -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1855927 Title: _poll_unconfirmed_resizes may not retry later if confirm_resize fails in API Status in OpenStack Compute (nova): New Bug description: This is based on code inspection but let's say I have configured my computes to set resize_confirm_window=3600 to automatically confirm a resized server after 1 hour. Within that hour, let's say the source compute service is down. The periodic task gets the unconfirmed migrations with status='finished' which have been updated some time older than the given configurable window: https://github.com/openstack/nova/blob/5a3ef39539ca112ae0552aef5cbd536338db61b7/nova/compute/manager.py#L8793 https://github.com/openstack/nova/blob/5a3ef39539ca112ae0552aef5cbd536338db61b7/nova/db/sqlalchemy/api.py#L4342 The periodic task then calls the compute API code to confirm the resize: https://github.com/openstack/nova/blob/c295e395d/nova/compute/manager.py#L7160 which changes the migration status to 'confirming': https://github.com/openstack/nova/blob/5a3ef39539ca112ae0552aef5cbd536338db61b7/nova/compute/api.py#L3684 And casts off to the source compute: https://github.com/openstack/nova/blob/5a3ef39539ca112ae0552aef5cbd536338db61b7/nova/compute/rpcapi.py#L600 Now if the source compute is down and that fails, the compute manager task code will handle it and say it will retry later: https://github.com/openstack/nova/blob/c295e395d/nova/compute/manager.py#L7163 However, because the migration status was changed from 'finished' to 'confirming' the task will not retry because it won't find the migration given the DB query. And trying to confirm the resize via the API will fail as well because we'll get MigrationNotFoundByStatus since the migration status is no longer 'finished': https://github.com/openstack/nova/blob/5a3ef39539ca112ae0552aef5cbd536338db61b7/nova/compute/api.py#L3681 The compute manager code should probably mark the migration status as 'finished' again if it's really going to try later, or mark the migration status as 'error'. Note that the confirm_resize method in the compute manager doesn't mark the migration status as 'error' if something fails there either: https://github.com/openstack/nova/blob/c295e395d/nova/compute/manager.py#L3807 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1855927/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1855875] Re: When create a new server instance this error occurred.
This looks like a configuration issue. What is the value of your transport_url config option in both the nova config and cell_mappings table? ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1855875 Title: When create a new server instance this error occurred. Status in OpenStack Compute (nova): Invalid Bug description: 2019-12-10 17:17:31.679 31059 ERROR oslo.messaging._drivers.impl_rabbit [req-a5d14694-ebda-449d-9030-0998fc27eb3e c035f19ef5af4a108d5a3704f59362ba b52c7945d86f428e8cf16a4d886f1f9a - default default] Failed to publish message to topic 'nova': 'NoneType' object has no attribute '__getitem__' 2019-12-10 17:17:31.679 31059 ERROR oslo.messaging._drivers.impl_rabbit [req-a5d14694-ebda-449d-9030-0998fc27eb3e c035f19ef5af4a108d5a3704f59362ba b52c7945d86f428e8cf16a4d886f1f9a - default default] Unable to connect to AMQP server on 192.168.0.204:5672 after inf tries: 'NoneType' object has no attribute '__getitem__' 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi [req-a5d14694-ebda-449d-9030-0998fc27eb3e c035f19ef5af4a108d5a3704f59362ba b52c7945d86f428e8cf16a4d886f1f9a - default default] Unexpected exception in API method: MessageDeliveryFailure: Unable to connect to AMQP server on 192.168.0.204:5672 after inf tries: 'NoneType' object has no attribute '__getitem__' 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 671, in wrapped 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 686, in create 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi **create_kwargs) 2019-12-10 17:17:31.680 31059 ERROR nova.api.openstack.wsgi File
[Yahoo-eng-team] [Bug 1854235] Re: Missing image type 'ploop'
You're also going to need a nova change to update the virt driver interface so the libvirt driver reports the trait. ** Also affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1854235 Title: Missing image type 'ploop' Status in OpenStack Compute (nova): New Status in os-traits: Fix Committed Bug description: Description of the problem == Missing image type 'ploop' Recreate steps 1. upload image: openstack image creat --file xxx.img --disk-format ploop --container-format bare --public test-ploop 2. configure: [scheduler]/query_placement_for_image_type_support=true, then restart devstack@n-sch 3. create server: openstack server create --flavor flv-cpu --image test-ploop --network private vm-test Expected: The server created fail. Actual: server created successfully. Traits of compute node: # openstack --os-placement-api-version 1.6 resource provider trait list 98029e51-2f81-46a3-9746-443174a1ba51 +---+ | name | +---+ | HW_CPU_HYPERTHREADING | | COMPUTE_IMAGE_TYPE_RAW| | HW_CPU_X86_SSE2 | | HW_CPU_X86_SVM| | HW_CPU_X86_SSE| | COMPUTE_IMAGE_TYPE_ISO| | COMPUTE_VOLUME_ATTACH_WITH_TAG| | COMPUTE_VOLUME_EXTEND | | HW_CPU_X86_MMX| | COMPUTE_VOLUME_MULTI_ATTACH | | COMPUTE_NET_ATTACH_INTERFACE_WITH_TAG | | COMPUTE_IMAGE_TYPE_AKI| | COMPUTE_IMAGE_TYPE_ARI| | COMPUTE_IMAGE_TYPE_AMI| | COMPUTE_DEVICE_TAGGING| | COMPUTE_IMAGE_TYPE_QCOW2 | | COMPUTE_NET_ATTACH_INTERFACE | | COMPUTE_TRUSTED_CERTS | | CUSTOM_LICENSED_WINDOWS | +---+ Version of os-traits and Nova === os-traits: # pip show os-traits Name: os-traits Version: 1.1.0 nova: # git log commit ab6834145f3fe1d33ce7f292727a6bc2be50efd9 (HEAD -> stable/train, origin/stable/train) Merge: 66585e8af1 821506a50c Author: Zuul Date: Mon Oct 21 18:50:20 2019 + Merge "Fix exception translation when creating volume" into stable/train To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1854235/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1835926] Re: Volume attachment may fail after rescuing instance on an image with different hw_disk_bus
** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Changed in: nova/train Status: New => Fix Released ** Changed in: nova/stein Status: New => In Progress ** Changed in: nova/rocky Status: New => In Progress ** Changed in: nova/queens Status: New => In Progress ** Changed in: nova/train Assignee: (unassigned) => Alexandre arents (aarents) ** Changed in: nova/stein Assignee: (unassigned) => Alexandre arents (aarents) ** Changed in: nova/rocky Assignee: (unassigned) => Alexandre arents (aarents) ** Changed in: nova/queens Assignee: (unassigned) => Alexandre arents (aarents) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1835926 Title: Volume attachment may fail after rescuing instance on an image with different hw_disk_bus Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) queens series: In Progress Status in OpenStack Compute (nova) rocky series: In Progress Status in OpenStack Compute (nova) stein series: In Progress Status in OpenStack Compute (nova) train series: Fix Released Bug description: Description === Look likes rescue may update instances.root_device_name if rescue image has different disk bus (image property hw_disk_bus) than instance. This introduce a mimatch between device name and driver used for instance: During instance config generation, nova guess the disk bus driver according table instance_system_metadata.image_hw_disk_bus, and get root device name from table instances.root_device_name. Because of this mismatch, cinder attachment may failed with the following error message in compute log: unable to execute QEMU command 'device_add': Duplicate ID 'virtio-disk0' for device Probable solution is to avoid rescue action to update instance.root_device_name Steps to reproduce == On a fresh master devstack: openstack image save cirros-0.4.0-x86_64-disk --file /tmp/cirros-0.4.0-x86_64-disk.disk #create a new image, but an scsi one: openstack image create --container-format bare --disk-format qcow2 --file /tmp/cirros-0.4.0-x86_64-disk.disk --property hw_disk_bus='scsi' --property hw_scsi_model='virtio-scsi' cirros-0.4.0-x86_64-scsi-disk #create instance with default virtio driver: openstack server create --flavor m1.small --image cirros-0.4.0-x86_64-disk --nic net-id=private test mysql> select root_device_name from instances where uuid='xxx' /dev/vda #rescue instance but with the scsi image: $openstack server rescue --image cirros-0.4.0-x86_64-scsi-disk mysql> select root_device_name from instances where uuid='xxx' /dev/sda $openstack server unrescue # root_device_name is still on sda should be on vda according instance metadata mysql> select root_device_name from instances where uuid='xxx' /dev/sda $virsh dumpxml instance-0001 | grep "bus='virtio" # at the next hard reboot new xml is generated with scsi device name BUT with virtio driver. $openstack server reboot --hard xxx $virsh dumpxml instance-0001 | grep -A 1 "bus='virtio" $openstack volume create --size 10 test $openstack server add volume 1c9b1582-5fc7-417a-a8a0-387e8833731f 0621430c-b0d2-4cca-8868-f86f36f1ef29 $sudo journalctl -u devstack@n-cpu.service | grep Duplicate Jul 05 09:29:54 alex-devstack-compute2 nova-compute[28285]: ERROR nova.virt.libvirt.driver [None req-38714989-4deb-4a05-bdfc-3418edbda7e3 demo demo] [instance: 1c9b1582-5fc7-417a-a8a0-387e8833731f] Failed to attach volume at mountpoint: /dev/vda: libvirtError: internal error: unable to execute QEMU command 'device_add': Duplicate ID 'virtio-disk0' for device Error probably comes from the fact that nova lookup for next availiable virtio device based on name, which is vda - virtio-disk0 (as root device is currently sda) but because root device sda is already using virtio-disk0 it failed. Expected result === instance root_device_name should remain the same as before rescue/unrescue, regardless of image used for rescuing. Actual result = instance root_device_name is updated according the hw_disk_bus property for the image used during rescue(and never set back to original value) To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1835926/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help :
[Yahoo-eng-team] [Bug 1838392] Re: BDMNotFound raised and stale block devices left over when simultaneously reboot and deleting an instance
** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Changed in: nova/queens Status: New => In Progress ** Changed in: nova/rocky Status: New => In Progress ** Changed in: nova/stein Status: New => In Progress ** Changed in: nova/train Status: New => In Progress ** Changed in: nova/queens Assignee: (unassigned) => Lee Yarwood (lyarwood) ** Changed in: nova/rocky Assignee: (unassigned) => Lee Yarwood (lyarwood) ** Changed in: nova/stein Assignee: (unassigned) => Lee Yarwood (lyarwood) ** Changed in: nova/train Assignee: (unassigned) => Lee Yarwood (lyarwood) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1838392 Title: BDMNotFound raised and stale block devices left over when simultaneously reboot and deleting an instance Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) queens series: In Progress Status in OpenStack Compute (nova) rocky series: In Progress Status in OpenStack Compute (nova) stein series: In Progress Status in OpenStack Compute (nova) train series: In Progress Bug description: Description === Simultaneous requests to reboot and delete an instance _will_ race as only the call to delete takes a lock against the instance.uuid. One possible outcome of this seen in the wild with the Libvirt driver is that the request to soft reboot will eventually turn into a hard reboot, reconnecting volumes that the delete request has already disconnected. These volumes will eventually be unmapped on the Cinder side by the delete request leaving stale devices on the host. Additionally BDMNotFound is raised by the reboot operation as the delete operation has already deleted the BDMs. Steps to reproduce == $ nova reboot $instance && nova delete $instance Expected result === The instance reboots and is then deleted without any errors raised. Actual result = BDMNotFound raised and stale block devices left over. Environment === 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ 1599e3cf68779eafaaa2b13a273d3bebd1379c19 / 19.0.0.0rc1-992-g1599e3cf68 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? Libvirt + QEMU/kvm 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? N/A 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs == To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1838392/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1844568] Re: [compute] "create_test_server" if networks is undefined and more than one network is present
tempest.scenario.test_minbw_allocation_placement.MinBwAllocationPlacementTest.test_qos_min_bw_allocation_basic,
which was enabled in the nova-next job this week:
https://review.opendev.org/#/c/697180/
is likely causing the spike in failures in the nova-next job. See this
failed job:
https://zuul.opendev.org/t/openstack/build/b7872ac3886e4cc7a060c1376a2a3395/log
/job-output.txt#79124
The failing compute api tests are running while that test is going and
that scenario test creates a public network, so that's likely the cause
of the failure.
The nova-next job should probably blacklist that test until this bug is
fixed.
** Also affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1844568
Title:
[compute] "create_test_server" if networks is undefined and more than
one network is present
Status in OpenStack Compute (nova):
Triaged
Status in tempest:
In Progress
Bug description:
This problem was detected in "ServersNegativeTestJSON" [1]. When a
server is created ("cls.create_test_server"), if a network is not
defined but several networks are present in this project, Nova raises
the exception "NetworkAmbiguous", a seen in this log snippet:
Sep 13 23:44:15.520980 ubuntu-bionic-limestone-regionone-0011283625
devstack@n-api.service[27339]: DEBUG nova.network.neutronv2.api [None
req-c95ecec2-8d10-4984-8ba9-b608161dd645
tempest-ServersNegativeTestJSON-445859222
tempest-ServersNegativeTestJSON-445859222] validate_networks() for None
{{(pid=27340) validate_networks
/opt/stack/nova/nova/network/neutronv2/api.py:2251}}
Sep 13 23:44:15.754945 ubuntu-bionic-limestone-regionone-0011283625
devstack@n-api.service[27339]: INFO nova.api.openstack.wsgi [None
req-c95ecec2-8d10-4984-8ba9-b608161dd645
tempest-ServersNegativeTestJSON-445859222
tempest-ServersNegativeTestJSON-445859222] HTTP exception thrown: Multiple
possible networks found, use a Network ID to be more specific.
We can see that the network information provided to the server creation is
empty but Nova tries to assign a default single network for this server.
However Nova does not assign this default network because several networks are
present for this project ID. In this case, the server creation should be
specific passing the network information.
[1]https://58a87e825b9766115d07-cec36eea8e90c9127fc5a72b798cfeab.ssl.cf2.rackcdn.com/670177/9/check/networking-ovn-tempest-dsvm-ovs-release/b58638a/testr_results.html.gz
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1844568/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1855015] Re: Intermittent fails in nova-next job with "Multiple possible networks found, use a Network ID to be more specific."
*** This bug is a duplicate of bug 1844568 ***
https://bugs.launchpad.net/bugs/1844568
It looks like that tempest test is actually expecting the error:
https://opendev.org/openstack/tempest/src/branch/master/tempest/api/compute/servers/test_attach_interfaces.py#L236
try:
iface = self._test_create_interface(server)
except lib_exc.BadRequest as e:
msg = ('Multiple possible networks found, use a Network ID to be '
'more specific.')
if not CONF.compute.fixed_network_name and six.text_type(e) == msg:
raise
else:
ifs.append(iface)
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1855015
Title:
Intermittent fails in nova-next job with "Multiple possible networks
found, use a Network ID to be more specific."
Status in OpenStack Compute (nova):
New
Bug description:
There was something similar before [1] but it was 100% and in one job.
This is intermittent and in multiple jobs across multiple projects.
http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Multiple%20possible%20networks%20found,%20use%20a%20Network%20ID%20to%20be%20more%20specific%5C%22
[1] https://bugs.launchpad.net/nova/+bug/1822605
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1855015/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1855015] Re: Intermittent fails since 11/23 with "Multiple possible networks found, use a Network ID to be more specific."
*** This bug is a duplicate of bug 1844568 *** https://bugs.launchpad.net/bugs/1844568 Based on that query I guess you're specifically looking at the nova-next job, right? It looks like the most fails on that are from this series starting with https://review.opendev.org/#/c/697180/. Having said that, this is also hitting in the gate queue. Note that the 11/23 in the bug title is misleading since logstash only goes back 10 days for us so you're just seeing where it falls off. ** Summary changed: - Intermittent fails since 11/23 with "Multiple possible networks found, use a Network ID to be more specific." + Intermittent fails in nova-next job with "Multiple possible networks found, use a Network ID to be more specific." ** No longer affects: neutron -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1855015 Title: Intermittent fails in nova-next job with "Multiple possible networks found, use a Network ID to be more specific." Status in OpenStack Compute (nova): New Bug description: There was something similar before [1] but it was 100% and in one job. This is intermittent and in multiple jobs across multiple projects. http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Multiple%20possible%20networks%20found,%20use%20a%20Network%20ID%20to%20be%20more%20specific%5C%22 [1] https://bugs.launchpad.net/nova/+bug/1822605 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1855015/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1825018] Re: security group driver gets loaded way too much in the api
** Also affects: nova/ocata Importance: Undecided Status: New ** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/pike Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Changed in: nova/ocata Importance: Undecided => Low ** Changed in: nova/queens Importance: Undecided => Low ** Changed in: nova/pike Importance: Undecided => Low ** Changed in: nova/train Importance: Undecided => Low ** Changed in: nova/stein Importance: Undecided => Low ** Changed in: nova/rocky Importance: Undecided => Low ** Changed in: nova/ocata Status: New => Confirmed ** Changed in: nova/pike Status: New => Confirmed ** Changed in: nova/rocky Status: New => Confirmed ** Changed in: nova/train Status: New => Confirmed ** Changed in: nova/stein Status: New => Confirmed ** Changed in: nova/queens Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1825018 Title: security group driver gets loaded way too much in the api Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) ocata series: Confirmed Status in OpenStack Compute (nova) pike series: Confirmed Status in OpenStack Compute (nova) queens series: Confirmed Status in OpenStack Compute (nova) rocky series: Confirmed Status in OpenStack Compute (nova) stein series: Confirmed Status in OpenStack Compute (nova) train series: Confirmed Bug description: There was a fix in Mitaka https://review.openstack.org/#/c/256073/ to cache the security group driver once it was loaded per process. That cache was removed in Newton https://review.openstack.org/#/c/325684/. I put up a test patch to see how many times the security group driver gets loaded https://review.openstack.org/#/c/652783/ and in the neutron-grenade-multinode job the nova-api logs show it getting loaded over 1000 times (browser count hit tops out at 1000). So the fix from mitaka was definitely regressed in newton and we should add the driver cache code again. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1825018/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1853495] Re: Unable to create an instance based on an encrypted volume snapshot
** Also affects: nova/stein
Importance: Undecided
Status: New
** Also affects: nova/train
Importance: Undecided
Status: New
** Changed in: nova/train
Status: New => In Progress
** Changed in: nova/stein
Status: New => In Progress
** Changed in: nova/train
Assignee: (unassigned) => Lee Yarwood (lyarwood)
** Changed in: nova/stein
Assignee: (unassigned) => Lee Yarwood (lyarwood)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1853495
Title:
Unable to create an instance based on an encrypted volume snapshot
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) stein series:
In Progress
Status in OpenStack Compute (nova) train series:
In Progress
Bug description:
Description
===
Nova is currently unable to create and then boot from volumes from encrypted
volume snapshots when the original volume_type is not provided as part of the
block_device_mapping request to n-api. Failure to provide this will result in
the following error when attempting to launch the instance:
Details: {'code': 500, 'created': '2019-11-21T16:56:58Z', 'message':
'Build of instance 176ff71f-1dff-433b-92c7-5a15bade375f aborted:
Invalid input received: Invalid input received: Invalid volume_type
provided: 9142150e-70fe-4b14-942f-015025de4a68 (requested type is not
compatible; recommend omitting the type argument). (H'}"
Steps to reproduce
==
* Create an encrypted volume snapshot
* Attempt to create an instance based on this without providing a volume_type
in the block_device_mapping.
Expected result
===
Nova copies the original volume_type if one is not provided in the request.
Actual result
=
Nova fails to copy the original volume_type leading to a failure when
attempting to create a volume based on the encrypted volume snapshot as the
types do not match.
Environment
===
1. Exact version of OpenStack you are running. See the following
list for all releases: http://docs.openstack.org/releases/
master
2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?
Libvirt + KVM
2. Which storage type did you use?
(For example: Ceph, LVM, GPFS, ...)
What's the version of that?
N/A
3. Which networking type did you use?
(For example: nova-network, Neutron with OpenVSwitch, ...)
N/A
Logs & Configs
==
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1853495/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1783565] Re: ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host intermittently fails with "Instance compute service state on host2 expected to be down, but it was
** Also affects: nova/stein
Importance: Undecided
Status: New
** Also affects: nova/rocky
Importance: Undecided
Status: New
** Also affects: nova/train
Importance: Undecided
Status: New
** Changed in: nova/train
Status: New => Fix Released
** Changed in: nova/train
Importance: Undecided => Low
** Changed in: nova/rocky
Status: New => Confirmed
** Changed in: nova/rocky
Importance: Undecided => Low
** Changed in: nova/stein
Importance: Undecided => Low
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1783565
Title:
ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host
intermittently fails with "Instance compute service state on host2
expected to be down, but it was up."
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) rocky series:
Confirmed
Status in OpenStack Compute (nova) stein series:
New
Status in OpenStack Compute (nova) train series:
Fix Released
Bug description:
http://logs.openstack.org/32/584032/5/check/nova-tox-functional-
py35/7061ec1/job-output.txt.gz#_2018-07-25_03_16_46_462415
18-07-25 03:16:46.418499 | ubuntu-xenial | {5}
nova.tests.functional.test_server_group.ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host
[14.070214s] ... FAILED
2018-07-25 03:16:46.418582 | ubuntu-xenial |
2018-07-25 03:16:46.418645 | ubuntu-xenial | Captured traceback:
2018-07-25 03:16:46.418705 | ubuntu-xenial | ~~~
2018-07-25 03:16:46.418798 | ubuntu-xenial | b'Traceback (most recent
call last):'
2018-07-25 03:16:46.419095 | ubuntu-xenial | b' File
"/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/test_server_group.py",
line 456, in test_evacuate_with_anti_affinity_no_valid_host'
2018-07-25 03:16:46.419232 | ubuntu-xenial | b"
self.admin_api.post_server_action(servers[1]['id'], post)"
2018-07-25 03:16:46.419471 | ubuntu-xenial | b' File
"/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py",
line 294, in post_server_action'
2018-07-25 03:16:46.419602 | ubuntu-xenial | b"'/servers/%s/action' %
server_id, data, **kwargs).body"
2018-07-25 03:16:46.419841 | ubuntu-xenial | b' File
"/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py",
line 235, in api_post'
2018-07-25 03:16:46.419975 | ubuntu-xenial | b'return
APIResponse(self.api_request(relative_uri, **kwargs))'
2018-07-25 03:16:46.420187 | ubuntu-xenial | b' File
"/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py",
line 213, in api_request'
2018-07-25 03:16:46.420263 | ubuntu-xenial | b'response=response)'
2018-07-25 03:16:46.420545 | ubuntu-xenial |
b'nova.tests.functional.api.client.OpenStackApiException: Unexpected status
code: {"badRequest": {"message": "Compute service of host2 is still in use.",
"code": 400}}'
2018-07-25 03:16:46.420581 | ubuntu-xenial | b''
2018-07-25 03:16:46.420606 | ubuntu-xenial |
2018-07-25 03:16:46.420654 | ubuntu-xenial | Captured stderr:
2018-07-25 03:16:46.420702 | ubuntu-xenial |
2018-07-25 03:16:46.421102 | ubuntu-xenial |
b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350:
OsloDBDeprecationWarning: EngineFacade is deprecated; please use
oslo_db.sqlalchemy.enginefacade'
2018-07-25 03:16:46.421240 | ubuntu-xenial | b' self._legacy_facade =
LegacyEngineFacade(None, _factory=self)'
2018-07-25 03:16:46.421623 | ubuntu-xenial |
b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350:
OsloDBDeprecationWarning: EngineFacade is deprecated; please use
oslo_db.sqlalchemy.enginefacade'
2018-07-25 03:16:46.421751 | ubuntu-xenial | b' self._legacy_facade =
LegacyEngineFacade(None, _factory=self)'
2018-07-25 03:16:46.422054 | ubuntu-xenial |
b"/home/zuul/src/git.openstack.org/openstack/nova/nova/test.py:323:
DeprecationWarning: Using class 'MoxStubout' (either directly or via
inheritance) is deprecated in version '3.5.0'"
2018-07-25 03:16:46.422174 | ubuntu-xenial | b' mox_fixture =
self.useFixture(moxstubout.MoxStubout())'
2018-07-25 03:16:46.422537 | ubuntu-xenial |
b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/paste/deploy/loadwsgi.py:22:
DeprecationWarning: Parameters to load are deprecated. Call .resolve and
.require separately.'
2018-07-25 03:16:46.422664 | ubuntu-xenial | b' return
pkg_resources.EntryPoint.parse("x=" + s).load(False)'
2018-07-25 03:16:46.422928 | ubuntu-xenial |
[Yahoo-eng-team] [Bug 1853926] Re: Failed to build docs cause of InvocationError
Clearly we're not having problems building docs and this can't be triaged with the actual doc build logs and the error. Did you have local changes that might have caused the docs build to fail? If so, investigate the log for errors. Do you have stale pyc files locally? If so, do: find . -name *.pyc -delete And then try again. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1853926 Title: Failed to build docs cause of InvocationError Status in OpenStack Compute (nova): Invalid Bug description: Description === When i use `tox -edocs`, got error ERROR: InvocationError for command /home/src/bug_1853745/.tox/docs/bin/sphinx-build -W --keep-going -b html -d doc/build/doctrees doc/source doc/build/html (exited with code 1) summary _ ERROR: docs: commands failed Environment === git log commit 3ead7d00a58c445fee8403ef3df41eec586b250d (origin/master, origin/HEAD, gerrit/master) Merge: 12e0c04dc0 83baeaa9f2 Author: Zuul Date: Sun Nov 24 00:31:49 2019 + Merge "Remove nova-manage network, floating commands" To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1853926/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1551703] Re: Resize a vm that vm_state is "stopped" failure， vm's task_state rollback to "active"
** Also affects: nova/train Importance: Undecided Status: New ** Changed in: nova/train Importance: Undecided => Low ** Changed in: nova/train Status: New => In Progress ** Changed in: nova/train Assignee: (unassigned) => Matt Riedemann (mriedem) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1551703 Title: Resize a vm that vm_state is "stopped" failure， vm's task_state rollback to "active" Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) train series: In Progress Bug description: 1. version kilo 2015.1.0 2. Reproduce steps 2.1 create a instance, then stop it. [root@SBCJNailSlot3 ~(keystone_admin)]# nova list +--++-+--+-+---+ | ID | Name | Status | Task State | Power State | Networks | +--++-+--+-+---+ | 6fe59445-fb89-47ab-9ead-2476c4522a61 | njq| SHUTOFF | -| Shutdown| test=192.168.1.52 | +--++-+--+-+---+ 2.2 resize the instance use a new flavor which it's disk less than current flavor's disk [root@SBCJNailSlot3 ~(keystone_admin)]# nova resize 6fe59445-fb89-47ab-9ead-2476c4522a61 45 disk value in the current flavor of instance “njq” is 20 disk value in the flavor which id equal 45 is 18. So this resize action will trigger ResizeError that msg is unable to resize disk down. Then enter the rollback process 2.3 rollback result： [root@SBCJNailSlot3 ~(keystone_admin)]# nova list +--++-+--+-+---+ | ID | Name | Status | Task State | Power State | Networks | +--++-+--+-+---+ | 6fe59445-fb89-47ab-9ead-2476c4522a61 | njq| ACTIVE | -| Shutdown| test=192.168.1.52 | +--++-+--+-+---+ Although the finally vm_state of instance will be set to stoped by heal_instance_state. But the process often takes some time. IMO， This process is not reasonable, and need fix. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1551703/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1853800] Re: Nova-compute generates incorrect exit code in case if child spawning failed
This is likely something that is controlled down in the oslo.service library code which nova-compute uses. ** Also affects: oslo.service Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1853800 Title: Nova-compute generates incorrect exit code in case if child spawning failed Status in OpenStack Compute (nova): New Status in oslo.service: New Bug description: In situation if nova-compute main process fails to spawn a child (worker) process e.g. because of database connection failure: ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) worker process exits with code 1, however, exit code of main process is 0. It shold be 1 because the service failed to start because of error. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1853800/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1853370] [NEW] resize_claim lazy-loads at least 3 joined fields in separate DB calls
Public bug reported: During a resize_claim the ResourceTracker lazy-loads in 3 separate calls to the DB (over RPC) these 3 fields: b"2019-11-20 16:13:29,521 DEBUG [nova.objects.instance] Lazy-loading 'pci_requests' on Instance uuid c0fdac69-b360-4526-917e-16fb018cb8a3" b"2019-11-20 16:13:29,525 DEBUG [nova.objects.instance] Lazy-loading 'resources' on Instance uuid c0fdac69-b360-4526-917e-16fb018cb8a3" b"2019-11-20 16:13:29,527 DEBUG [nova.objects.instance] Lazy-loading 'pci_devices' on Instance uuid c0fdac69-b360-4526-917e-16fb018cb8a3" It seems we should be able to collapse that into a single DB call to load the necessary fields in a single call. We could add a new extra_attrs kwarg to the Instance.refresh method so we can keep using the same instance we have in memory (and is shared by the ComputeManager method calling the resize_claim) or we could add a new load_if_not_present() method to the Instance object. I'm not sure if there are pros/cons either way on using refresh or adding a new method. ** Affects: nova Importance: Low Status: Confirmed ** Tags: performance resize ** Changed in: nova Status: New => Confirmed ** Changed in: nova Importance: Undecided => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1853370 Title: resize_claim lazy-loads at least 3 joined fields in separate DB calls Status in OpenStack Compute (nova): Confirmed Bug description: During a resize_claim the ResourceTracker lazy-loads in 3 separate calls to the DB (over RPC) these 3 fields: b"2019-11-20 16:13:29,521 DEBUG [nova.objects.instance] Lazy-loading 'pci_requests' on Instance uuid c0fdac69-b360-4526-917e-16fb018cb8a3" b"2019-11-20 16:13:29,525 DEBUG [nova.objects.instance] Lazy-loading 'resources' on Instance uuid c0fdac69-b360-4526-917e-16fb018cb8a3" b"2019-11-20 16:13:29,527 DEBUG [nova.objects.instance] Lazy-loading 'pci_devices' on Instance uuid c0fdac69-b360-4526-917e-16fb018cb8a3" It seems we should be able to collapse that into a single DB call to load the necessary fields in a single call. We could add a new extra_attrs kwarg to the Instance.refresh method so we can keep using the same instance we have in memory (and is shared by the ComputeManager method calling the resize_claim) or we could add a new load_if_not_present() method to the Instance object. I'm not sure if there are pros/cons either way on using refresh or adding a new method. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1853370/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1853280] Re: nova-live-migration job constantly fails on stable/pike
This might be fixed by https://review.opendev.org/#/c/695191/. ** Also affects: devstack-plugin-ceph Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1853280 Title: nova-live-migration job constantly fails on stable/pike Status in devstack-plugin-ceph: New Status in OpenStack Compute (nova): Invalid Status in OpenStack Compute (nova) pike series: New Bug description: signature: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22E%3A%20Unable%20to%20locate%20package%20python3-cephfs%5C%22 example: https://zuul.opendev.org/t/openstack/build/0a199eeccc334b98a2eaf67998eef8b5/log/job-output.txt#5821 It seems that the devstack-plugin-ceph install fails as it tries to install py3 packages that are not available in the package mirror. I think the merge of https://review.opendev.org/#/c/694330/ in devstack-plugin-ceph triggering the fault To manage notifications about this bug go to: https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1853280/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1853166] Re: nova not installable on py2 environments which breaks the gate
** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1853166 Title: nova not installable on py2 environments which breaks the gate Status in OpenStack Compute (nova): Fix Released Bug description: As noted in the revert https://review.opendev.org/#/c/694891/ change https://review.opendev.org/#/c/687954/ made nova not installable on environments running python < 3.6: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22nova%20requires%20Python%20'%3E%3D3.6'%20but%20the%20running%20Python%20is%5C%22%20AND%20tags%3A%5C%22console%5C%22=7d Since devstack isn't by default using py3 lots of downstream jobs broke in other projects. This bug is to track the fix: https://review.opendev.org/#/c/695007/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1853166/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1853166] [NEW] nova not installable on py2 environments which breaks the gate
Public bug reported: As noted in the revert https://review.opendev.org/#/c/694891/ change https://review.opendev.org/#/c/687954/ made nova not installable on environments running python < 3.6: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22nova%20requires%20Python%20'%3E%3D3.6'%20but%20the%20running%20Python%20is%5C%22%20AND%20tags%3A%5C%22console%5C%22=7d Since devstack isn't by default using py3 lots of downstream jobs broke in other projects. This bug is to track the fix: https://review.opendev.org/#/c/695007/ ** Affects: nova Importance: Critical Assignee: Luigi Toscano (ltoscano) Status: In Progress ** Tags: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1853166 Title: nova not installable on py2 environments which breaks the gate Status in OpenStack Compute (nova): In Progress Bug description: As noted in the revert https://review.opendev.org/#/c/694891/ change https://review.opendev.org/#/c/687954/ made nova not installable on environments running python < 3.6: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22nova%20requires%20Python%20'%3E%3D3.6'%20but%20the%20running%20Python%20is%5C%22%20AND%20tags%3A%5C%22console%5C%22=7d Since devstack isn't by default using py3 lots of downstream jobs broke in other projects. This bug is to track the fix: https://review.opendev.org/#/c/695007/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1853166/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852993] Re: Don't delete compute node when deleting service other than nova-compute
Deleting a nova-consoleauth service shouldn't have anything to do with deleting compute_nodes records, only deleting nova-compute services, but I guess this doesn't filter on the service binary being nova-compute: https://github.com/openstack/nova/blob/a054d03adef692db22e2466084e50cbf50112bb0/nova/db/sqlalchemy/api.py#L415 However, a nova-consoleauth service id shouldn't be mapped to a compute node record, but that's probably where the OR is breaking things - the service_id is likely NULL and the host is the same. ** Changed in: nova Importance: Undecided => Medium ** Tags added: db ** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Changed in: nova/rocky Status: New => Confirmed ** Changed in: nova/stein Status: New => Confirmed ** Changed in: nova/stein Importance: Undecided => Medium ** Changed in: nova/train Importance: Undecided => Medium ** Changed in: nova/rocky Importance: Undecided => Medium ** Changed in: nova/train Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1852993 Title: Don't delete compute node when deleting service other than nova- compute Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) rocky series: Confirmed Status in OpenStack Compute (nova) stein series: Confirmed Status in OpenStack Compute (nova) train series: Confirmed Bug description: When upgrading to Stein, nova-consoleauth service is deprecated and should be removed. However if nova-consoleauth service is located on the same host with nova-compute, matching row in compute_nodes table is soft-deleted as well, making nova-compute service report in log, that stale resource provider exists in placement: 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager [req-f0255008-c398-406c-bca0-12cdc34fc0b4 - - - - -] Error updating resources for node vzstor1.vstoragedomain.: ResourceProviderCreationFailed: Failed to create resource provider vzstor1.vstoragedomain 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager Traceback (most recent call last): 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7399, in update_available_resource_for_node 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager rt.update_available_resource(context, nodename) 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update_available_resource(context, resources) 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return f(*args, **kwargs) 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._init_compute_node(context, resources) 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 562, in _init_compute_node 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager self._update(context, cn) 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 887, in _update 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager inv_data, 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid, 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs) 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 1106, in set_inventory_for_provider 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 667, in _ensure_resource_provider 2019-11-18 16:03:20.069 7 ERROR nova.compute.manager
[Yahoo-eng-team] [Bug 1853089] Re: openstack env in Stein, CLI or Dashboard response slowly
This is going to require profiling on your part. This isn't a nova bug. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1853089 Title: openstack env in Stein, CLI or Dashboard response slowly Status in neutron: New Status in OpenStack Compute (nova): Invalid Bug description: Installed Stein OpenStack via Kolla, OpenStack node info as below: 3 * Controller node which running in Virtualbox --- each machine memory: 32G and Disk:300G 2 * Compute node which is physical rack server after operating in some days, the OpenStack response become very slow, the dashboard response is more slower. for example: run "openstack port list --timing" in CLI, it would spent about 13s when got 60 port data, and dashboard would spent more than 20s. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1853089/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852665] Re: HostMappingNotFound in wigi.py
*** This bug is a duplicate of bug 1780727 *** https://bugs.launchpad.net/bugs/1780727 What release are you using? You must not have this fix: https://review.opendev.org/#/q/I0d7644db3537a67b94e75972b3c4fce25a623763 I'm going to mark this as a duplicate of bug 1780727. ** This bug has been marked a duplicate of bug 1780727 Handle HostMappingNotFound when deleting a service -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1852665 Title: HostMappingNotFound in wigi.py Status in OpenStack Compute (nova): New Bug description: It seems to be diffrent from #1780727. # openstack compute service delete 26 Failed to delete compute service with ID '26': 发生意外 API 错误。请在 http://bugs.launchpad.net/nova/ 处报告此错误，并且附上 Nova API 日志（如果可能）。 (HTTP 500) (Request-ID: req-a693a664-5834-48c6-acd9-c92d8b2eea28) nova-api-log: 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi [req-099cb32c-dfbb-4eaf-a067-a3737720977f 7c177645d83f4a769d51ab4b9801a90d 04627ab3503248cb8d714ac1a44f543f - default default] Unexpected exception in API method: HostMappingNotFound: \u4e3b\u673a 'computer3713hpdl580'\u6ca1\u6709\u6620\u5c04\u5230\u4efb\u4f55\u5355\u5143 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 237, in delete 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi service.host) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi function_name, call_dict, binary, tb) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi self.force_reraise() 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi six.reraise(self.type_, self.value, self.tb) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi return f(self, context, *args, **kw) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 5416, in remove_host_from_aggregate 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi mapping = objects.HostMapping.get_by_host(context, host_name) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi result = fn(cls, context, *args, **kwargs) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/objects/host_mapping.py", line 100, in get_by_host 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi db_mapping = cls._get_by_host_from_db(context, host) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 993, in wrapper 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi return fn(*args, **kwargs) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/objects/host_mapping.py", line 95, in _get_by_host_from_db 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi raise exception.HostMappingNotFound(name=host) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi HostMappingNotFound: \u4e3b\u673a 'computer3713hpdl580'\u6ca1\u6709\u6620\u5c04\u5230\u4efb\u4f55\u5355\u5143 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi 。 # openstack compute service list |egrep "(3713|ID)" | ID | Binary | Host| Zone | Status | State | Updated At | | 26 | nova-compute | computer3713hpdl580 | sugonZone | disabled | down | 2019-07-24T01:25:20.00 | # nova service-list |egrep "(3713|Id)" | Id | Binary | Host | Zone | Status | State | Updated_at
[Yahoo-eng-team] [Bug 1837199] Re: nova-manage Tracebeck on missing arg
Something must have regressed in Queens because going back to Pike this
still works:
usage: nova-manage db [-h]
{archive_deleted_rows,ironic_flavor_migration,null_instance_uuid_scan,online_data_migrations,sync,version}
...
nova-manage db: error: too few arguments
I don't see anything obvious in the nova-manage code in Queens that
changed so it must have been something in oslo.config.
** Also affects: nova/train
Importance: Undecided
Status: New
** Also affects: nova/stein
Importance: Undecided
Status: New
** Also affects: nova/queens
Importance: Undecided
Status: New
** Changed in: nova/train
Importance: Undecided => Low
** Changed in: nova/queens
Status: New => Confirmed
** Changed in: nova/queens
Importance: Undecided => Low
** Changed in: nova/stein
Status: New => Confirmed
** Changed in: nova/train
Status: New => Confirmed
** Changed in: nova/stein
Importance: Undecided => Low
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1837199
Title:
nova-manage Tracebeck on missing arg
Status in OpenStack Compute (nova):
In Progress
Status in OpenStack Compute (nova) queens series:
Confirmed
Status in OpenStack Compute (nova) stein series:
Confirmed
Status in OpenStack Compute (nova) train series:
Confirmed
Bug description:
# nova-manage cell_v2
An error has occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/oslo_config/cfg.py", line
3179, in __getattr__
return getattr(self._conf._namespace, name)
AttributeError: '_Namespace' object has no attribute 'action_fn'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/stack/nova/nova/cmd/manage.py", line 2205, in main
fn, fn_args, fn_kwargs = cmd_common.get_action_fn()
File "/opt/stack/nova/nova/cmd/common.py", line 169, in get_action_fn
fn = CONF.category.action_fn
File "/usr/local/lib/python3.7/site-packages/oslo_config/cfg.py", line
3181, in __getattr__
raise NoSuchOptError(name)
oslo_config.cfg.NoSuchOptError: no such option action_fn in group [DEFAULT]
# nova-manage cell_v2 help
usage: nova-manage cell_v2 [-h]
{create_cell,delete_cell,delete_host,discover_hosts,list_cells,list_hosts,map_cell0,map_cell_and_hosts,map_instances,simple_cell_setup,update_cell,verify_instance}
...
nova-manage cell_v2: error: argument action: invalid choice: 'help' (choose
from 'create_cell', 'delete_cell', 'delete_host', 'discover_hosts',
'list_cells', 'list_hosts', 'map_cell0', 'map_cell_and_hosts', 'map_instances',
'simple_cell_setup', 'update_cell', 'verify_instance')
# nova-manage cell_v2 -h
usage: nova-manage cell_v2 [-h]
{create_cell,delete_cell,delete_host,discover_hosts,list_cells,list_hosts,map_cell0,map_cell_and_hosts,map_instances,simple_cell_setup,update_cell,verify_instance}
...
positional arguments:
{create_cell,delete_cell,delete_host,discover_hosts,list_cells,list_hosts,map_cell0,map_cell_and_hosts,map_instances,simple_cell_setup,update_cell,verify_instance}
optional arguments:
-h, --helpshow this help message and exit
python version:
/usr/bin/python3 --version
Python 3.7.3
nova version:
$ git log -1
commit 78f9961d293e3b3e0ac62345b78abb1c9e2bd128 (HEAD -> master,
origin/master, origin/HEAD)
oslo.config 6.11.0
Instead of printing Traceback, nova-manage should give a hint for the
user choices.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1837199/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1837199] Re: nova-manage Tracebeck on missing arg
** No longer affects: oslo.config
** Changed in: nova
Assignee: (unassigned) => Matt Riedemann (mriedem)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1837199
Title:
nova-manage Tracebeck on missing arg
Status in OpenStack Compute (nova):
Confirmed
Bug description:
# nova-manage cell_v2
An error has occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/oslo_config/cfg.py", line
3179, in __getattr__
return getattr(self._conf._namespace, name)
AttributeError: '_Namespace' object has no attribute 'action_fn'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/stack/nova/nova/cmd/manage.py", line 2205, in main
fn, fn_args, fn_kwargs = cmd_common.get_action_fn()
File "/opt/stack/nova/nova/cmd/common.py", line 169, in get_action_fn
fn = CONF.category.action_fn
File "/usr/local/lib/python3.7/site-packages/oslo_config/cfg.py", line
3181, in __getattr__
raise NoSuchOptError(name)
oslo_config.cfg.NoSuchOptError: no such option action_fn in group [DEFAULT]
# nova-manage cell_v2 help
usage: nova-manage cell_v2 [-h]
{create_cell,delete_cell,delete_host,discover_hosts,list_cells,list_hosts,map_cell0,map_cell_and_hosts,map_instances,simple_cell_setup,update_cell,verify_instance}
...
nova-manage cell_v2: error: argument action: invalid choice: 'help' (choose
from 'create_cell', 'delete_cell', 'delete_host', 'discover_hosts',
'list_cells', 'list_hosts', 'map_cell0', 'map_cell_and_hosts', 'map_instances',
'simple_cell_setup', 'update_cell', 'verify_instance')
# nova-manage cell_v2 -h
usage: nova-manage cell_v2 [-h]
{create_cell,delete_cell,delete_host,discover_hosts,list_cells,list_hosts,map_cell0,map_cell_and_hosts,map_instances,simple_cell_setup,update_cell,verify_instance}
...
positional arguments:
{create_cell,delete_cell,delete_host,discover_hosts,list_cells,list_hosts,map_cell0,map_cell_and_hosts,map_instances,simple_cell_setup,update_cell,verify_instance}
optional arguments:
-h, --helpshow this help message and exit
python version:
/usr/bin/python3 --version
Python 3.7.3
nova version:
$ git log -1
commit 78f9961d293e3b3e0ac62345b78abb1c9e2bd128 (HEAD -> master,
origin/master, origin/HEAD)
oslo.config 6.11.0
Instead of printing Traceback, nova-manage should give a hint for the
user choices.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1837199/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852846] Re: Unexpected API Error
Well there are obviously problems on the controllers because there are DB connection errors in the logs. You could try using --os-compute-api-version 2.34 to avoid the RPC timeout but that kind of error is generally symptomatic of bigger problems, like failures connecting to the conductor or scheduler service, or the scheduler taking too long, or the DB being down, etc. This isn't a bug, it's a support request so I'm going to invalidate it. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1852846 Title: Unexpected API Error Status in OpenStack Compute (nova): Invalid Bug description: Hello, I cannot live migrate to another new compute because of the following error: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. (HTTP 500) (Request-ID: req-2f4a47c3-e46f-405b-bb5c-56796732a04d) Please help me to fix it, many thanks. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1852846/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852665] Re: HostMappingNotFound in wigi.py
Which release is nova in the API? As you can see from the service listing, nova CLI does it with the latest microversion negotiated between novaclient and the server so you see the service id as a uuid (microversion 2.53+) but openstack compute service list uses microversion 2.1 by default, so you see the int id. I think the issue is you're trying to delete the service using openstack client but with the integer id and starting in pike you need to use the uuid to uniquely identify the service in a cell. See the note here: https://docs.openstack.org/python-openstackclient/latest/cli/command- objects/compute-service.html#compute-service-delete "If using --os-compute-api-version 2.53 or greater, the ID is a UUID which can be retrieved by listing compute services using the same 2.53+ microversion." Anyway, try deleting the service using microversion 2.53 and the service id as a uuid and see if that solves the issue. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1852665 Title: HostMappingNotFound in wigi.py Status in OpenStack Compute (nova): Invalid Bug description: It seems to be diffrent from #1780727. # openstack compute service delete 26 Failed to delete compute service with ID '26': 发生意外 API 错误。请在 http://bugs.launchpad.net/nova/ 处报告此错误，并且附上 Nova API 日志（如果可能）。 (HTTP 500) (Request-ID: req-a693a664-5834-48c6-acd9-c92d8b2eea28) nova-api-log: 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi [req-099cb32c-dfbb-4eaf-a067-a3737720977f 7c177645d83f4a769d51ab4b9801a90d 04627ab3503248cb8d714ac1a44f543f - default default] Unexpected exception in API method: HostMappingNotFound: \u4e3b\u673a 'computer3713hpdl580'\u6ca1\u6709\u6620\u5c04\u5230\u4efb\u4f55\u5355\u5143 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 237, in delete 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi service.host) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi function_name, call_dict, binary, tb) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi self.force_reraise() 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi six.reraise(self.type_, self.value, self.tb) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi return f(self, context, *args, **kw) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 5416, in remove_host_from_aggregate 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi mapping = objects.HostMapping.get_by_host(context, host_name) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi result = fn(cls, context, *args, **kwargs) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/objects/host_mapping.py", line 100, in get_by_host 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi db_mapping = cls._get_by_host_from_db(context, host) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 993, in wrapper 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi return fn(*args, **kwargs) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/objects/host_mapping.py", line 95, in _get_by_host_from_db 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi raise exception.HostMappingNotFound(name=host) 2019-11-15 08:41:23.576 11235 ERROR nova.api.openstack.wsgi HostMappingNotFound:
[Yahoo-eng-team] [Bug 1852610] Re: API allows source compute service/node deletion while instances are pending a resize confirm/revert
This goes back further than Rocky but since Queens is in extended
maintenance mode upstream I figure it's best to just focus on Rocky+ for
now.
** Changed in: nova
Assignee: (unassigned) => Matt Riedemann (mriedem)
** Also affects: nova/rocky
Importance: Undecided
Status: New
** Also affects: nova/train
Importance: Undecided
Status: New
** Also affects: nova/stein
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1852610
Title:
API allows source compute service/node deletion while instances are
pending a resize confirm/revert
Status in OpenStack Compute (nova):
Triaged
Status in OpenStack Compute (nova) rocky series:
New
Status in OpenStack Compute (nova) stein series:
New
Status in OpenStack Compute (nova) train series:
New
Bug description:
This is split off from bug 1829479 which is about deleting a compute
service which had servers evacuated from it which will orphan resource
providers in placement.
A similar scenario is true where the API will allow deleting a source
compute service which has migration-based allocations for the source
node resource provider and pending instance resizes involving the
source node. A simple scenario is:
1. create a server on host1
2. resize or cold migrate it to a dest host2
3. delete the compute service for host1
At this point the resource provider for host1 is orphaned.
4. try to confirm/revert the resize of the server which will fail
because the compute node for host1 is gone and this results in the
server going to ERROR status
Based on the discussion in this mailing list thread:
http://lists.openstack.org/pipermail/openstack-
discuss/2019-November/010843.html
We should probably have the DELETE /os-services/{service_id} API block
trying to delete a service that has pending migrations.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1852610/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852610] [NEW] API allows source compute service/node deletion while instances are pending a resize confirm/revert
Public bug reported:
This is split off from bug 1829479 which is about deleting a compute
service which had servers evacuated from it which will orphan resource
providers in placement.
A similar scenario is true where the API will allow deleting a source
compute service which has migration-based allocations for the source
node resource provider and pending instance resizes involving the source
node. A simple scenario is:
1. create a server on host1
2. resize or cold migrate it to a dest host2
3. delete the compute service for host1
At this point the resource provider for host1 is orphaned.
4. try to confirm/revert the resize of the server which will fail
because the compute node for host1 is gone and this results in the
server going to ERROR status
Based on the discussion in this mailing list thread:
http://lists.openstack.org/pipermail/openstack-
discuss/2019-November/010843.html
We should probably have the DELETE /os-services/{service_id} API block
trying to delete a service that has pending migrations.
** Affects: nova
Importance: Medium
Status: Triaged
** Tags: api placement resize
** Changed in: nova
Status: New => Triaged
** Changed in: nova
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1852610
Title:
API allows source compute service/node deletion while instances are
pending a resize confirm/revert
Status in OpenStack Compute (nova):
Triaged
Bug description:
This is split off from bug 1829479 which is about deleting a compute
service which had servers evacuated from it which will orphan resource
providers in placement.
A similar scenario is true where the API will allow deleting a source
compute service which has migration-based allocations for the source
node resource provider and pending instance resizes involving the
source node. A simple scenario is:
1. create a server on host1
2. resize or cold migrate it to a dest host2
3. delete the compute service for host1
At this point the resource provider for host1 is orphaned.
4. try to confirm/revert the resize of the server which will fail
because the compute node for host1 is gone and this results in the
server going to ERROR status
Based on the discussion in this mailing list thread:
http://lists.openstack.org/pipermail/openstack-
discuss/2019-November/010843.html
We should probably have the DELETE /os-services/{service_id} API block
trying to delete a service that has pending migrations.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1852610/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852609] [NEW] Reboot an instance in nova - split the rescue part out
Public bug reported: - [x] This doc is inaccurate in this way: I didn't think nova had any documentation about the rescue operation but found that it's hidden in the reboot user guide: https://docs.openstack.org/nova/latest/user/reboot.html Ironically that page has more documentation about rescue than it does about reboot. We should likely split the rescue docs out into a separate user guide reference page. --- Release: on 2018-10-23 11:25:36 SHA: 552ab225896c6697cb7f6f582693be8588daa13c Source: https://opendev.org/openstack/nova/src/doc/source/user/reboot.rst URL: https://docs.openstack.org/nova/latest/user/reboot.html ** Affects: nova Importance: Undecided Status: New ** Tags: doc low-hanging-fruit ** Tags added: low-hanging-fruit -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1852609 Title: Reboot an instance in nova - split the rescue part out Status in OpenStack Compute (nova): New Bug description: - [x] This doc is inaccurate in this way: I didn't think nova had any documentation about the rescue operation but found that it's hidden in the reboot user guide: https://docs.openstack.org/nova/latest/user/reboot.html Ironically that page has more documentation about rescue than it does about reboot. We should likely split the rescue docs out into a separate user guide reference page. --- Release: on 2018-10-23 11:25:36 SHA: 552ab225896c6697cb7f6f582693be8588daa13c Source: https://opendev.org/openstack/nova/src/doc/source/user/reboot.rst URL: https://docs.openstack.org/nova/latest/user/reboot.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1852609/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1852465] Re: ExternalNetworkAttachForbidden should result in BuildAbortException, not reschedule
This goes back a long time but I'll just target to the non-extended- maintenance branches. ** Also affects: nova/rocky Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/train Importance: Undecided Status: New ** Changed in: nova/rocky Status: New => Confirmed ** Changed in: nova/train Status: New => Confirmed ** Changed in: nova/rocky Importance: Undecided => Low ** Changed in: nova/stein Status: New => Confirmed ** Changed in: nova/train Importance: Undecided => Low ** Changed in: nova/stein Importance: Undecided => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1852465 Title: ExternalNetworkAttachForbidden should result in BuildAbortException, not reschedule Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) rocky series: Confirmed Status in OpenStack Compute (nova) stein series: Confirmed Status in OpenStack Compute (nova) train series: Confirmed Bug description: I saw this in a CI run where creating a server on an external network as a non-admin user failed and was rescheduled. Here are the failures from the compute logs: https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n-cpu.txt.gz#28774 https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/compute1/logs/screen-n-cpu.txt.gz#37415 Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [None req-41117798-8a4e-469f-bfbb-8bdfdea1a83f demo demo] [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Instance failed to spawn: nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0 Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Traceback (most recent call last): Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 2659, in _build_resources Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] yield resources Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 2433, in _build_and_run_instance Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] block_device_info=block_device_info) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3467, in spawn Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] mdevs=mdevs) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6019, in _get_guest_xml Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] network_info_str = str(network_info) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 601, in __str__ Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] return self._sync_wrapper(fn, *args, **kwargs) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 584, in _sync_wrapper Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] self.wait() Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 616, in wait Nov 13 15:35:44.312293
[Yahoo-eng-team] [Bug 1852468] Re: network router:external value is non-boolean (Internal) which causes server create failure
Nevermind about the value, it's openstack CLI that is translating the false value to "Internal": https://github.com/openstack/python- openstackclient/blob/d17a1c8039807cdac29e77eb5f0724d181bdd831/openstackclient/network/v2/network.py#L33 ** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1852468 Title: network router:external value is non-boolean (Internal) which causes server create failure Status in neutron: Invalid Bug description: I noticed this here: https://review.opendev.org/#/c/693248/ Nova has this post-test script: https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/console#4/0/0/controller Which creates an internal private network: + /opt/stack/nova/gate/post_test_hook.sh:main:120 : openstack network create net0 --provider-network-type vlan --provider-physical-network public --provider-segment 100 --project demo +---+--+ | Field | Value | +---+--+ | admin_state_up| UP | | availability_zone_hints | | | availability_zones| | | created_at| 2019-11-13T15:35:09Z | | description | | | dns_domain| None | | id| c0c449d7-267d-42f3-86a1-b27b472edf65 | | ipv4_address_scope| None | | ipv6_address_scope| None | | is_default| False | | is_vlan_transparent | None | | location | cloud='', project.domain_id=, project.domain_name=, project.id='5e899e65a6fc46ba9e8bf4210c4a6a2e', project.name=, region_name='RegionOne', zone= | | mtu | 1450 | | name | net0 | | port_security_enabled | True | | project_id| 5e899e65a6fc46ba9e8bf4210c4a6a2e | | provider:network_type | vlan | | provider:physical_network | public | | provider:segmentation_id | 100
[Yahoo-eng-team] [Bug 1852468] [NEW] network router:external value is non-boolean (Internal) which causes server create failure
Public bug reported: I noticed this here: https://review.opendev.org/#/c/693248/ Nova has this post-test script: https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/console#4/0/0/controller Which creates an internal private network: + /opt/stack/nova/gate/post_test_hook.sh:main:120 : openstack network create net0 --provider-network-type vlan --provider-physical-network public --provider-segment 100 --project demo +---+--+ | Field | Value | +---+--+ | admin_state_up| UP | | availability_zone_hints | | | availability_zones| | | created_at| 2019-11-13T15:35:09Z | | description | | | dns_domain| None | | id| c0c449d7-267d-42f3-86a1-b27b472edf65 | | ipv4_address_scope| None | | ipv6_address_scope| None | | is_default| False | | is_vlan_transparent | None | | location | cloud='', project.domain_id=, project.domain_name=, project.id='5e899e65a6fc46ba9e8bf4210c4a6a2e', project.name=, region_name='RegionOne', zone= | | mtu | 1450 | | name | net0 | | port_security_enabled | True | | project_id| 5e899e65a6fc46ba9e8bf4210c4a6a2e | | provider:network_type | vlan | | provider:physical_network | public | | provider:segmentation_id | 100 | | qos_policy_id | None | | revision_number | 1 | | router:external | Internal | | segments | None
[Yahoo-eng-team] [Bug 1852465] [NEW] ExternalNetworkAttachForbidden should result in BuildAbortException, not reschedule
Public bug reported: I saw this in a CI run where creating a server on an external network as a non-admin user failed and was rescheduled. Here are the failures from the compute logs: https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n-cpu.txt.gz#28774 https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/compute1/logs/screen-n-cpu.txt.gz#37415 Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [None req-41117798-8a4e-469f-bfbb-8bdfdea1a83f demo demo] [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Instance failed to spawn: nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0 Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Traceback (most recent call last): Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 2659, in _build_resources Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] yield resources Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 2433, in _build_and_run_instance Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] block_device_info=block_device_info) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3467, in spawn Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] mdevs=mdevs) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6019, in _get_guest_xml Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] network_info_str = str(network_info) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 601, in __str__ Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] return self._sync_wrapper(fn, *args, **kwargs) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 584, in _sync_wrapper Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] self.wait() Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 616, in wait Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] self[:] = self._gt.wait() Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 181, in wait Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] return self._exit_event.wait() Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/usr/local/lib/python3.6/dist-packages/eventlet/event.py", line 132, in wait Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] current.throw(*self._exc) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File
[Yahoo-eng-team] [Bug 1852458] [NEW] "create" instance action not created when instance is buried in cell0
Public bug reported:
Before cell0 was introduced the API would create the "create" instance
action for each instance in the nova cell database before casting off to
conductor to do scheduling:
https://github.com/openstack/nova/blob/mitaka-
eol/nova/compute/api.py#L1180
Note that conductor failed to "complete" the action with a failure
event:
https://github.com/openstack/nova/blob/mitaka-
eol/nova/conductor/manager.py#L374
But at least the action was created.
Since then, with cell0, if scheduling fails the instance is buried in
the cell0 database but no instance action is created. To illustrate, I
disabled the single nova-compute service on my devstack host and created
a server which failed with NoValidHost:
$ openstack server show build-fail1 -f value -c fault
{u'message': u'No valid host was found. ', u'code': 500, u'created':
u'2019-11-13T15:57:13Z'}
When listing instance actions I expected to see a "create" action but
there were none:
$ nova instance-action-list 008a7d52-dd83-4f52-a720-b3cfcc498259
+++-+++
| Action | Request_ID | Message | Start_Time | Updated_At |
+++-+++
+++-+++
This is because the "create" action is only created when the instance is
scheduled to a specific cell:
https://github.com/openstack/nova/blob/20.0.0/nova/conductor/manager.py#L1460
Solution:
The ComputeTaskManager._bury_in_cell0 method should also create a
"create" action in cell0 like it does for the instance BDMs and tags.
This goes back to Ocata: https://review.opendev.org/#/c/319379/
** Affects: nova
Importance: Low
Assignee: Matt Riedemann (mriedem)
Status: Triaged
** Affects: nova/ocata
Importance: Low
Status: Triaged
** Affects: nova/pike
Importance: Low
Status: Triaged
** Affects: nova/queens
Importance: Low
Status: Triaged
** Affects: nova/rocky
Importance: Low
Status: Triaged
** Affects: nova/stein
Importance: Low
Status: Triaged
** Affects: nova/train
Importance: Low
Status: Triaged
** Tags: cells regression
** Also affects: nova/train
Importance: Undecided
Status: New
** Also affects: nova/pike
Importance: Undecided
Status: New
** Also affects: nova/rocky
Importance: Undecided
Status: New
** Also affects: nova/queens
Importance: Undecided
Status: New
** Also affects: nova/ocata
Importance: Undecided
Status: New
** Also affects: nova/stein
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1852458
Title:
"create" instance action not created when instance is buried in cell0
Status in OpenStack Compute (nova):
Triaged
Status in OpenStack Compute (nova) ocata series:
Triaged
Status in OpenStack Compute (nova) pike series:
Triaged
Status in OpenStack Compute (nova) queens series:
Triaged
Status in OpenStack Compute (nova) rocky series:
Triaged
Status in OpenStack Compute (nova) stein series:
Triaged
Status in OpenStack Compute (nova) train series:
Triaged
Bug description:
Before cell0 was introduced the API would create the "create" instance
action for each instance in the nova cell database before casting off
to conductor to do scheduling:
https://github.com/openstack/nova/blob/mitaka-
eol/nova/compute/api.py#L1180
Note that conductor failed to "complete" the action with a failure
event:
https://github.com/openstack/nova/blob/mitaka-
eol/nova/conductor/manager.py#L374
But at least the action was created.
Since then, with cell0, if scheduling fails the instance is buried in
the cell0 database but no instance action is created. To illustrate, I
disabled the single nova-compute service on my devstack host and
created a server which failed with NoValidHost:
$ openstack server show build-fail1 -f value -c fault
{u'message': u'No valid host was found. ', u'code': 500, u'created':
u'2019-11-13T15:57:13Z'}
When listing instance actions I expected to see a "create" action but
there were none:
$ nova instance-action-list 008a7d52-dd83-4f52-a720-b3cfcc498259
+++-+++
| Action | Request_ID | Message | Start_Time | Updated_At |
+++-+++
+++-+++
This is because the "create" action is only created when the instance
is scheduled to a specific cell:
https://github.com/openstack/nova/blob/20.0.0/nova/conductor/manager.py#L1460
Solution:
The ComputeTaskManager._bury_in_cell0 method should also create
[Yahoo-eng-team] [Bug 1852446] [NEW] Hypervisors in nova - no subpage details for ironic
Public bug reported: - [x] This is a doc addition request. The admin configuration for hypervisors does not have a subpage with details about configuring nova with the ironic compute driver. There are at least a few things that could go into a page like that: * Summary of what it does and how it interacts with the ironic service as a 'hypervisor'. Some of that information is available in the ironic docs, e.g.: https://docs.openstack.org/ironic/latest/install/get_started.html?highlight=nova #interaction-with-openstack-components Since it comes up from time to time, I would also mention that the ironic driver is the only one in nova where the compute_nodes table record is 1:M with the compute services table record for the given host, meaning a nova-compute service can manage multiple ComputeNodes, and the ComputeNodes for the ironic driver managed compute service uses the ironic node uuid for the compute node hypervisor_hostname (nodename) and uuid fields. And ironic node : compute node : instance are 1:1:1. This is more contributor/reference information but it's worth mentioning somewhere since it's kind of tribal knowledge in nova. * Ironic-specific configuration: https://docs.openstack.org/nova/latest/configuration/config.html#ironic - This could also include things like configuring baremetal flavors: https://docs.openstack.org/ironic/latest/install/configure-nova- flavors.html - Running multiple nova-computes in HA mode managing the same set of nodes: https://specs.openstack.org/openstack/nova- specs/specs/newton/implemented/ironic-multiple-compute-hosts.html * Scaling and performance issues. Some of this is discussed in this mailing list thread: http://lists.openstack.org/pipermail/openstack- discuss/2019-November/thread.html#10655 - Partitioning schemes: https://specs.openstack.org/openstack/nova- specs/specs/stein/implemented/ironic-conductor-groups.html * Known limitations / missing features, e.g. move operations (migrate/resize). --- Release: on 2018-09-04 18:11:45 SHA: 8a71962e0149fa9ad7f66c17849bf69df3e78d33 Source: https://opendev.org/openstack/nova/src/doc/source/admin/configuration/hypervisors.rst URL: https://docs.openstack.org/nova/latest/admin/configuration/hypervisors.html ** Affects: nova Importance: Undecided Status: New ** Tags: doc ironic ** Description changed: - [x] This is a doc addition request. The admin configuration for hypervisors does not have a subpage with details about configuring nova with the ironic compute driver. There are at least a few things that could go into a page like that: * Summary of what it does and how it interacts with the ironic service as a 'hypervisor'. Some of that information is available in the ironic docs, e.g.: https://docs.openstack.org/ironic/latest/install/get_started.html?highlight=nova #interaction-with-openstack-components Since it comes up from time to time, I would also mention that the - ironic driver is the only one in nova would the compute_nodes table + ironic driver is the only one in nova where the compute_nodes table record is 1:M with the compute services table record for the given host, meaning a nova-compute service can manage multiple ComputeNodes, and the ComputeNodes for the ironic driver managed compute service uses the ironic node uuid for the compute node hypervisor_hostname (nodename) and uuid fields. And ironic node : compute node : instance are 1:1:1. This is more contributor/reference information but it's worth mentioning somewhere since it's kind of tribal knowledge in nova. * Ironic-specific configuration: https://docs.openstack.org/nova/latest/configuration/config.html#ironic - This could also include things like configuring baremetal flavors: https://docs.openstack.org/ironic/latest/install/configure-nova- flavors.html - Running multiple nova-computes in HA mode managing the same set of nodes: https://specs.openstack.org/openstack/nova- specs/specs/newton/implemented/ironic-multiple-compute-hosts.html * Scaling and performance issues. Some of this is discussed in this mailing list thread: http://lists.openstack.org/pipermail/openstack- discuss/2019-November/thread.html#10655 - Partitioning schemes: https://specs.openstack.org/openstack/nova- specs/specs/stein/implemented/ironic-conductor-groups.html * Known limitations / missing features, e.g. move operations (migrate/resize). --- Release: on 2018-09-04 18:11:45 SHA: 8a71962e0149fa9ad7f66c17849bf69df3e78d33 Source: https://opendev.org/openstack/nova/src/doc/source/admin/configuration/hypervisors.rst URL: https://docs.openstack.org/nova/latest/admin/configuration/hypervisors.html -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova).
[Yahoo-eng-team] [Bug 1852207] Re: reschedule ignores requested availability zone
** Changed in: nova Status: New => Invalid ** Tags added: pike-only -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1852207 Title: reschedule ignores requested availability zone Status in OpenStack Compute (nova): Invalid Status in OpenStack Compute (nova) pike series: In Progress Bug description: This is only observed on stable/pike. Reproduce - * Create an aggregate mapped to an availability zone * Add one host to the aggregate * Make sure (e.g. with fault injection) that the nova-compute on the host in the aggregate will fail during the instance_claim process to trigger re-schedule * boot an instance requesting the az Expected Instance boot fails with NoValidHost Actual -- Instance is rescheduled and the scheduler ignores the requested az and therefore instance is booted outside of the requested az. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1852207/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1714235] Re: os-migrateLive API does not restrict one from trying to migrate to the original host
I'm just going to invalidate this since if you're using microversion < 2.34 you'll get a 400 because of the UnableToMigrateToSelf error from conductor and if you're using >= 2.34 you'll get a 202 response but the operation will fail and a fault will be recorded, there would just be more work to do, e.g. swapping allocations, calling the scheduler, reverting the allocation swap, etc. As long as the instance isn't put into ERROR state though we're probably OK. ** Changed in: nova Assignee: jichenjc (jichenjc) => (unassigned) ** Changed in: nova Status: In Progress => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1714235 Title: os-migrateLive API does not restrict one from trying to migrate to the original host Status in OpenStack Compute (nova): Invalid Bug description: This is purely based on code inspection, but the compute API method 'evacuate' does not check if the specified host (if there was one) is different from instance.host. It checks if the service is up on that host, which could be down and you can still specify the instance.host. Eventually the compute API will RPC cast to conductor task manager which will fail with an RPC error trying to RPC cast to the ComputeManager.rebuild_instance method on the compute service, which is down. The bug here is that instead of getting an obvious 400 error from the API, you're left with not much for details when it fails. There should be an instance action and finish event, but only the admin can see the traceback in the event. Also, the instance.task_state would be left in 'rebuilding' state, and would require it to be reset to use the instance again. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1714235/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1732363] Re: Use subprocess securely
I'm going to invalidate this because the code being pointed out is just dev/test scripts, not production runtime code. It can be fixed if someone wants to restore and rework the patch but it's super low priority IMO. ** Changed in: nova Importance: Undecided => Low ** Changed in: nova Status: In Progress => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1732363 Title: Use subprocess securely Status in OpenStack Compute (nova): Invalid Bug description: Due to https://docs.openstack.org/bandit/latest/plugins/subprocess_popen_with_shell_equals_true.html, and reference https://security.openstack.org/guidelines/dg_avoid-shell-true.html and https://security.openstack.org/guidelines/dg_use-subprocess-securely.html, we should use python pipes to avoid shells and use subprocess more securely. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1732363/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1851871] [NEW] Unit tests intermittently fail with "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" during service destroy
Public bug reported:
Seen here:
https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e8/691900/3/check
/openstack-tox-py27/6e8b215/testr_results.html.gz
ft1.34:
nova.tests.unit.conductor.test_conductor.ConductorTaskAPITestCase.test_unshelve_instance_schedule_and_rebuild_novalid_hosttesttools.testresult.real._StringException:
pythonlogging:'': {{{
2019-11-08 16:07:22,725 INFO [nova.virt.driver] Loading compute driver
'fake.SmallFakeDriver'
2019-11-08 16:07:22,735 INFO [nova.compute.manager] Deleting orphan compute
node 2 hypervisor host is fake_phyp1, nodes are set(['fakenode1', 'fakenode2'])
2019-11-08 16:07:22,739 ERROR [nova.compute.manager] Failed to delete compute
node resource provider for compute node 906973a2-d73c-4254-8f1b-dfce1ae55359:
An auth plugin is required to determine endpoint URL
2019-11-08 16:07:22,753 WARNING [nova.compute.resource_tracker] No compute node
record for fake-mini:fakenode1
2019-11-08 16:07:22,760 INFO [nova.compute.resource_tracker] Compute node
record created for fake-mini:fakenode1 with uuid:
f4a97ca1-8388-4005-a3e0-d714beebcd69
2019-11-08 16:07:22,803 WARNING [nova.compute.resource_tracker] No compute node
record for fake-mini:fakenode2
2019-11-08 16:07:22,807 INFO [nova.compute.resource_tracker] Compute node
record created for fake-mini:fakenode2 with uuid:
bbd1fd88-6b11-4de7-9d0e-a909d51df43d
2019-11-08 16:07:22,851 INFO [nova.service] Starting conductor node (version
20.1.0)
2019-11-08 16:07:22,970 WARNING [nova.conductor.manager] No valid host found
for unshelve instance
}}}
Traceback (most recent call last):
File
"/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/fixture.py",
line 125, in cleanUp
return self._cleanups(raise_errors=raise_first)
File
"/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/callmany.py",
line 89, in __call__
reraise(error[0], error[1], error[2])
File
"/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/callmany.py",
line 83, in __call__
cleanup(*args, **kwargs)
File "nova/service.py", line 271, in kill
self.service_ref.destroy()
File
"/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
line 226, in wrapper
return fn(self, *args, **kwargs)
File "nova/objects/service.py", line 446, in destroy
db.service_destroy(self._context, self.id)
File "nova/db/api.py", line 99, in service_destroy
return IMPL.service_destroy(context, service_id)
File "nova/db/sqlalchemy/api.py", line 222, in wrapped
with ctxt_mgr.writer.using(context):
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File
"/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py",
line 1064, in _transaction_scope
context=context) as resource:
File
"/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py",
line 710, in _produce_block
self._writer()
File
"/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py",
line 725, in _writer
"Can't upgrade a READER transaction "
TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction
But there are a few hits for this in other tests/jobs for unrelated
changes:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TypeError%3A%20Can't%20upgrade%20a%20READER%20transaction%20to%20a%20WRITER
%20mid-
transaction%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20project%3A%5C%22openstack%2Fnova%5C%22=7d
12 fails in 7 days.
** Affects: nova
Importance: Undecided
Status: New
** Tags: gate-failure testing
** Summary changed:
- Functional tests intermittently fail with "TypeError: Can't upgrade a READER
transaction to a WRITER mid-transaction" during service destroy
+ Unit tests intermittently fail with "TypeError: Can't upgrade a READER
transaction to a WRITER mid-transaction" during service destroy
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1851871
Title:
Unit tests intermittently fail with "TypeError: Can't upgrade a READER
transaction to a WRITER mid-transaction" during service destroy
Status in OpenStack Compute (nova):
New
Bug description:
Seen here:
https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e8/691900/3/check
/openstack-tox-py27/6e8b215/testr_results.html.gz
ft1.34:
[Yahoo-eng-team] [Bug 1794827] Re: Queens openstack client does not support --block-device
** Also affects: nova/pike Importance: Undecided Status: New ** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1794827 Title: Queens openstack client does not support --block-device Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: New Status in OpenStack Compute (nova) queens series: New Status in OpenStack Compute (nova) rocky series: In Progress Bug description: This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes: - [x] This doc is inaccurate in this way: Anchor #create-volume-from-image-and-boot-instance describes the usage of option --block-device to generate a volume from an image and boot from there. That option is not implemented in the Queens openstack client, although it does exist in the nova client. I suggest substituting the openstack client syntax with the equivalent nova syntax. If you have a troubleshooting or support issue, use the following resources: - Ask OpenStack: http://ask.openstack.org - The mailing list: http://lists.openstack.org - IRC: 'openstack' channel on Freenode --- Release: 17.0.6.dev116 on 2018-09-22 00:26 SHA: 8157b16a65094035cbf0efc75e7bd6a2feee150f Source: https://git.openstack.org/cgit/openstack/nova/tree/doc/source/user/launch-instance-from-volume.rst URL: https://docs.openstack.org/nova/queens/user/launch-instance-from-volume.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1794827/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1783565] Re: ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host intermittently fails with "Instance compute service state on host2 expected to be down, but it was
gibi has a fix here: https://review.opendev.org/692252
** Changed in: nova
Assignee: (unassigned) => Balazs Gibizer (balazs-gibizer)
** Changed in: nova
Status: Invalid => In Progress
** Changed in: nova
Importance: Undecided => Low
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1783565
Title:
ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host
intermittently fails with "Instance compute service state on host2
expected to be down, but it was up."
Status in OpenStack Compute (nova):
In Progress
Bug description:
http://logs.openstack.org/32/584032/5/check/nova-tox-functional-
py35/7061ec1/job-output.txt.gz#_2018-07-25_03_16_46_462415
18-07-25 03:16:46.418499 | ubuntu-xenial | {5}
nova.tests.functional.test_server_group.ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host
[14.070214s] ... FAILED
2018-07-25 03:16:46.418582 | ubuntu-xenial |
2018-07-25 03:16:46.418645 | ubuntu-xenial | Captured traceback:
2018-07-25 03:16:46.418705 | ubuntu-xenial | ~~~
2018-07-25 03:16:46.418798 | ubuntu-xenial | b'Traceback (most recent
call last):'
2018-07-25 03:16:46.419095 | ubuntu-xenial | b' File
"/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/test_server_group.py",
line 456, in test_evacuate_with_anti_affinity_no_valid_host'
2018-07-25 03:16:46.419232 | ubuntu-xenial | b"
self.admin_api.post_server_action(servers[1]['id'], post)"
2018-07-25 03:16:46.419471 | ubuntu-xenial | b' File
"/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py",
line 294, in post_server_action'
2018-07-25 03:16:46.419602 | ubuntu-xenial | b"'/servers/%s/action' %
server_id, data, **kwargs).body"
2018-07-25 03:16:46.419841 | ubuntu-xenial | b' File
"/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py",
line 235, in api_post'
2018-07-25 03:16:46.419975 | ubuntu-xenial | b'return
APIResponse(self.api_request(relative_uri, **kwargs))'
2018-07-25 03:16:46.420187 | ubuntu-xenial | b' File
"/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/api/client.py",
line 213, in api_request'
2018-07-25 03:16:46.420263 | ubuntu-xenial | b'response=response)'
2018-07-25 03:16:46.420545 | ubuntu-xenial |
b'nova.tests.functional.api.client.OpenStackApiException: Unexpected status
code: {"badRequest": {"message": "Compute service of host2 is still in use.",
"code": 400}}'
2018-07-25 03:16:46.420581 | ubuntu-xenial | b''
2018-07-25 03:16:46.420606 | ubuntu-xenial |
2018-07-25 03:16:46.420654 | ubuntu-xenial | Captured stderr:
2018-07-25 03:16:46.420702 | ubuntu-xenial |
2018-07-25 03:16:46.421102 | ubuntu-xenial |
b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350:
OsloDBDeprecationWarning: EngineFacade is deprecated; please use
oslo_db.sqlalchemy.enginefacade'
2018-07-25 03:16:46.421240 | ubuntu-xenial | b' self._legacy_facade =
LegacyEngineFacade(None, _factory=self)'
2018-07-25 03:16:46.421623 | ubuntu-xenial |
b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_db/sqlalchemy/enginefacade.py:350:
OsloDBDeprecationWarning: EngineFacade is deprecated; please use
oslo_db.sqlalchemy.enginefacade'
2018-07-25 03:16:46.421751 | ubuntu-xenial | b' self._legacy_facade =
LegacyEngineFacade(None, _factory=self)'
2018-07-25 03:16:46.422054 | ubuntu-xenial |
b"/home/zuul/src/git.openstack.org/openstack/nova/nova/test.py:323:
DeprecationWarning: Using class 'MoxStubout' (either directly or via
inheritance) is deprecated in version '3.5.0'"
2018-07-25 03:16:46.422174 | ubuntu-xenial | b' mox_fixture =
self.useFixture(moxstubout.MoxStubout())'
2018-07-25 03:16:46.422537 | ubuntu-xenial |
b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/paste/deploy/loadwsgi.py:22:
DeprecationWarning: Parameters to load are deprecated. Call .resolve and
.require separately.'
2018-07-25 03:16:46.422664 | ubuntu-xenial | b' return
pkg_resources.EntryPoint.parse("x=" + s).load(False)'
2018-07-25 03:16:46.422928 | ubuntu-xenial |
b"/home/zuul/src/git.openstack.org/openstack/nova/nova/db/sqlalchemy/api.py:205:
DeprecationWarning: Property 'async_compat' has moved to 'function.async_'"
2018-07-25 03:16:46.423038 | ubuntu-xenial | b' reader_mode =
get_context_manager(context).async'
2018-07-25 03:16:46.423301 | ubuntu-xenial |
b"/home/zuul/src/git.openstack.org/openstack/nova/nova/db/sqlalchemy/api.py:205:
DeprecationWarning: Property 'async_compat' has moved to 'function.async_'"
2018-07-25
[Yahoo-eng-team] [Bug 1850735] Re: time.sleep(10) in server group func tests
*** This bug is a duplicate of bug 1783565 *** https://bugs.launchpad.net/bugs/1783565 ** This bug has been marked a duplicate of bug 1783565 ServerGroupTestV21.test_evacuate_with_anti_affinity_no_valid_host intermittently fails with "Instance compute service state on host2 expected to be down, but it was up." -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1850735 Title: time.sleep(10) in server group func tests Status in OpenStack Compute (nova): In Progress Bug description: In the server group functional tests in nova/tests/functional/test_server_group.py there are several places where time.sleep(10) is called in order to make sure a stopped compute service is considered "down" by the nova compute API. Instead of sleeping, we can set the service as "forced_down" to get the desired "down" compute service status and avoid unnecessary delays in these tests. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1850735/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1850280] Re: rejecting move operations with qos ports does not work if the operation is called with non admin user
** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Changed in: nova/stein Status: New => Triaged ** Changed in: nova/stein Importance: Undecided => Medium ** Changed in: nova/train Status: New => Triaged ** Changed in: nova/train Importance: Undecided => Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1850280 Title: rejecting move operations with qos ports does not work if the operation is called with non admin user Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) stein series: Triaged Status in OpenStack Compute (nova) train series: Triaged Bug description: At the start of the move operation the nova-api checks that the server has ports with resource request as some of the move operations are not supported for such servers yet. Unfortunately if move the operation is called by a non-admin user (either it is a resize, or another move operation with explicit policy change) then nova uses the non-admin token to query neutron. If the neutron port is queried with a non admin token then neutron does not return the resource_request to nova in the port response. Therefore nova thinks that the port ha no resource request and allows the operation. Reproduce in Ussuri === * Boot a server with qos port. * Change the nova policy to allow evacuate to be called by the owner of the server "os_compute_api:os-evacuate": "rule:admin_or_owner" * stop the nova-compute service on the host where the server currently running and wait until the controller decides that the compute is done * with the non-admin owner initiate the evacuate of the server Expected: * evacuate rejected Actual: * evacuate accepted (and later fail due to missing implementation) Triage == Due to [1] not using an admin client nova does not get the resource requests of the attached ports. Affected versions and operations The resize, migrate, live migrate, evacuate, unshelve opertaions are affected on master, Train, Stein. [1] https://github.com/openstack/nova/blob/9742a64403c0a0ae5e0b37df5b0bf3ba14ac4626/nova/api/openstack/common.py#L576 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1850280/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1849657] Re: allocation key is missing from the binding:profile of the neutron qos port when the server is created by a non-admin user
** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Changed in: nova/stein Status: New => Triaged ** Changed in: nova/train Status: New => Triaged ** Changed in: nova/stein Importance: Undecided => Medium ** Changed in: nova/train Importance: Undecided => Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1849657 Title: allocation key is missing from the binding:profile of the neutron qos port when the server is created by a non-admin user Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) stein series: Triaged Status in OpenStack Compute (nova) train series: Triaged Bug description: Description === When a server is create by a non-admin tenant with a qos neutron port Nova does not add the allocation key to the binding:profile of the port. Steps to reproduce == 1) Set up a devstack with bandwidth inventory * sudo ovs-vsctl add-br br-test * devstack local conf: [[post-config|/etc/neutron/neutron.conf]] [DEFAULT] service_plugins = router, placement, qos [[post-config|/etc/neutron/plugins/ml2/ml2_conf.ini]] [ml2] extension_drivers = port_security,qos mechanism_drivers = openvswitch tenant_network_types = vxlan [ml2_type_vlan] network_vlan_ranges = physnet0:1000:2000 [ovs] bridge_mappings = public:br-ex,physnet0:br-test resource_provider_bandwidths = br-test:5000:5000 [ovs_driver] vnic_type_blacklist = direct * stack.sh 2) As admin user set up a network and a qos policy: * openstack network create net-demo --provider-network-type vlan --provider-physical-network physnet0 --provider-segment 101 --share * openstack subnet create subnet-demo --network net-demo --subnet-range 10.0.4.0/24 * openstack network qos policy create qp-demo --share * openstack network qos rule create qp-demo --type minimum-bandwidth --min-kbps 1000 --egress * openstack network qos rule create qp-demo --type minimum-bandwidth --min-kbps 1000 --ingress 3) As a normal user (demo in devstack) create a port with the qos policy and create a server with the port * openstack port create port-normal-qos-demo --network net-demo --vnic-type normal --qos-policy qp-demo * openstack --os-compute-api-version 2.72 server create --image cirros-0.4.0-x86_64-disk --flavor c1 --nic port-id=port-normal-qos-demo vm-demo --wait Expected result === 1) Server is reaching ACTIVE state 2) Bandwidth allocation is created in placement according to the qp-demo policy 3) The allocation key of the binding:profile of the port-normal-qos-demo port contains the UUID of the placement resource provider from where the bandwidth resource is allocated from. Actual result = 1) and 2) are as expected but the binding:porfile of the neutron port does not have an allocation key. Note that if the server is booted as admin user then both 1) 2) 3) are as expected. Environment === Devstack from master: stack@aio:/opt/stack/nova$ git log --oneline | head -1 d3403e5294 Merge "Fix unit of hw_rng:rate_period" stack@aio:/opt/stack/neutron$ git log --oneline | head -1 2ffaa40b43 Merge "ovsdb monitor: handle modified ports" Triage == Looking at the port-normal-qos-demo port from the demo user. The resource_request filed of the port is None. While looking at the port from the admin user the resource_request field is properly filled according to the qos policy of the port. As demo: stack@aio:~$ openstack port show port-normal-qos-demo +-+-+ | Field | Value | +-+-+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | None
[Yahoo-eng-team] [Bug 1849695] Re: resize of server with qos ports fails if called by non admin user
** Also affects: nova/train Importance: Undecided Status: New ** Changed in: nova/train Status: New => Triaged -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1849695 Title: resize of server with qos ports fails if called by non admin user Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) train series: Triaged Bug description: As a non admin: * Create a server with a qos port. * Resize the server to another flavor. => Server goes to ERROR state and the following is logged in the nova- compute log on the destination host Oct 24 14:33:42 aio nova-compute[10293]: ERROR oslo_messaging.rpc.server PortUpdateFailed: Port update failed for port b1593c18-b088-4d5c-b3c6-bdd5348f3b52: Provider mappings are not available to the compute service but are required for ports with a resource request. Triage: Similarly to bug 1849657 Nova uses a non admin Neutron client to query the ports[1] at the start of the resize. The the resize operation is not called by an admin user then the resource_request field of the Neutron is not filled. This causes that Nova does allocate resources and does not create port - rp mapping for the qos ports on the destination node. But when the qos port is being updated on the destination host [2] nova uses an admin client and therefore sees the resource_request of the qos ports. As the port - rp mapping is missing for these ports the resize fails. [1] https://github.com/openstack/nova/blob/1bfa4626d13d0a73e63745cc4a864ae86d490daf/nova/network/neutronv2/api.py#L2228 [2] https://github.com/openstack/nova/blob/1bfa4626d13d0a73e63745cc4a864ae86d490daf/nova/network/neutronv2/api.py#L3305 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1849695/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1851430] Re: slow metadata performance with security groups that have a lot of rules
** Changed in: nova Assignee: Matt Riedemann (mriedem) => Doug Wiegley (dougwig) ** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/pike Importance: Undecided Status: New ** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Changed in: nova/pike Status: New => Confirmed ** Changed in: nova/queens Status: New => Confirmed ** Changed in: nova/rocky Status: New => Confirmed ** Changed in: nova/stein Status: New => Confirmed ** Changed in: nova/train Status: New => Confirmed ** Summary changed: - slow metadata performance with security groups that have a lot of rules + Slow metadata API performance with security groups that have a lot of rules ** Changed in: nova/pike Importance: Undecided => Medium ** Changed in: nova/stein Importance: Undecided => Medium ** Changed in: nova/queens Importance: Undecided => Medium ** Changed in: nova/rocky Importance: Undecided => Medium ** Changed in: nova/train Importance: Undecided => Medium ** Changed in: nova/pike Importance: Medium => Low ** Changed in: nova/queens Importance: Medium => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1851430 Title: Slow metadata API performance with security groups that have a lot of rules Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) pike series: Confirmed Status in OpenStack Compute (nova) queens series: Confirmed Status in OpenStack Compute (nova) rocky series: Confirmed Status in OpenStack Compute (nova) stein series: Confirmed Status in OpenStack Compute (nova) train series: Confirmed Bug description: This was reported here without a bug: https://review.opendev.org/#/c/656084/ The EC2 metadata API response includes a 'security-groups' key that is a list of security group names attached to the instance. The problem is for each security group attached to the instance, if the group has a lot of rules on it, it can be expensive to query (join) that information from neutron, especially if we don't care about the rules. By default, listing security groups includes the rules in the response: https://docs.openstack.org/api-ref/network/v2/index.html?expanded =list-security-groups-detail#list-security-groups For the purpose of the EC2 metadata API, we should just query security groups for their names. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1851430/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1851430] [NEW] slow metadata performance with security groups that have a lot of rules
Public bug reported: This was reported here without a bug: https://review.opendev.org/#/c/656084/ The EC2 metadata API response includes a 'security-groups' key that is a list of security group names attached to the instance. The problem is for each security group attached to the instance, if the group has a lot of rules on it, it can be expensive to query (join) that information from neutron, especially if we don't care about the rules. By default, listing security groups includes the rules in the response: https://docs.openstack.org/api-ref/network/v2/index.html?expanded=list- security-groups-detail#list-security-groups For the purpose of the EC2 metadata API, we should just query security groups for their names. ** Affects: nova Importance: Medium Status: Confirmed ** Tags: api metadata neutron performance ** Changed in: nova Importance: Undecided => Medium ** Changed in: nova Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1851430 Title: slow metadata performance with security groups that have a lot of rules Status in OpenStack Compute (nova): Confirmed Bug description: This was reported here without a bug: https://review.opendev.org/#/c/656084/ The EC2 metadata API response includes a 'security-groups' key that is a list of security group names attached to the instance. The problem is for each security group attached to the instance, if the group has a lot of rules on it, it can be expensive to query (join) that information from neutron, especially if we don't care about the rules. By default, listing security groups includes the rules in the response: https://docs.openstack.org/api-ref/network/v2/index.html?expanded =list-security-groups-detail#list-security-groups For the purpose of the EC2 metadata API, we should just query security groups for their names. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1851430/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1850694] Re: shelve doesn't handle UnexpectedTaskStateError
** Also affects: nova/rocky
Importance: Undecided
Status: New
** Also affects: nova/train
Importance: Undecided
Status: New
** Also affects: nova/queens
Importance: Undecided
Status: New
** Also affects: nova/stein
Importance: Undecided
Status: New
** Changed in: nova/queens
Status: New => In Progress
** Changed in: nova/stein
Status: New => In Progress
** Changed in: nova/rocky
Status: New => In Progress
** Changed in: nova/rocky
Importance: Undecided => Low
** Changed in: nova/rocky
Assignee: (unassigned) => Artom Lifshitz (notartom)
** Changed in: nova/queens
Assignee: (unassigned) => Artom Lifshitz (notartom)
** Changed in: nova/stein
Assignee: (unassigned) => Artom Lifshitz (notartom)
** Changed in: nova/train
Status: New => In Progress
** Changed in: nova/train
Importance: Undecided => Low
** Changed in: nova/train
Assignee: (unassigned) => Artom Lifshitz (notartom)
** Changed in: nova/stein
Importance: Undecided => Low
** Changed in: nova/queens
Importance: Undecided => Low
** Tags added: shelve
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1850694
Title:
shelve doesn't handle UnexpectedTaskStateError
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) queens series:
In Progress
Status in OpenStack Compute (nova) rocky series:
In Progress
Status in OpenStack Compute (nova) stein series:
In Progress
Status in OpenStack Compute (nova) train series:
In Progress
Bug description:
Description
===
Shelving a server expects its task state to be None. If it's not None
(for example, if attempting to shelve a server that's already being
shelved), we get a UnexpectedTaskStateError from the database layer
and return a 500 to the user. A 409 would be more appropriate.
Steps to reproduce
==
1. Send multiple shelve requests in quick succession.
Expected result
===
The initial request should be accepted, the rest should return 409.
Actual result
=
Error 500 for all requests except the first.
Environment
===
This was reported on OSP13 (Queens) originally [1].
Logs & Configs
==
2019-05-28 03:18:48.530 26 INFO nova.osapi_compute.wsgi.server
[req-1437e513-3e32-4243-8f5d-1a7e17c111df 3ff59a48497842e7a716a03a17e5bf8b
493b17f3b02b4f9ea6e71b1ae4c5ac5d - e4c6faf4dfb04f2da40c0595f1a424c7
e4c6faf4dfb04f2da40c0595f1a424c7] 10.101.4.137,10.101.4.1 "POST
/v2.1/493b17f3b02b4f9ea6e71b1ae4c5ac5d/servers/f905b880-9caa-465e-93c5-fffe9192c825/action
HTTP/1.1" status: 500 len: 622 time: 0.1237578
2019-05-28 03:18:48.529 26 INFO nova.api.openstack.wsgi
[req-1437e513-3e32-4243-8f5d-1a7e17c111df 3ff59a48497842e7a716a03a17e5bf8b
493b17f3b02b4f9ea6e71b1ae4c5ac5d - e4c6faf4dfb04f2da40c0595f1a424c7
e4c6faf4dfb04f2da40c0595f1a424c7] HTTP exception thrown: Unexpected API Error.
Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API
log if possible.
2019-05-28 03:18:48.529 26 DEBUG nova.api.openstack.wsgi
[req-1437e513-3e32-4243-8f5d-1a7e17c111df 3ff59a48497842e7a716a03a17e5bf8b
493b17f3b02b4f9ea6e71b1ae4c5ac5d - e4c6faf4dfb04f2da40c0595f1a424c7
e4c6faf4dfb04f2da40c0595f1a424c7] Returning 500 to user: Unexpected API Error.
Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API
log if possible.
__call__
/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1064
2019-05-28 03:18:48.523 26 ERROR nova.api.openstack.wsgi
[req-1437e513-3e32-4243-8f5d-1a7e17c111df 3ff59a48497842e7a716a03a17e5bf8b
493b17f3b02b4f9ea6e71b1ae4c5ac5d - e4c6faf4dfb04f2da40c0595f1a424c7
e4c6faf4dfb04f2da40c0595f1a424c7] Unexpected exception in API method:
UnexpectedTaskStateError: Conflict updating instance
f905b880-9caa-465e-93c5-fffe9192c825. Expected: {'task_state': [None]}. Actual:
{'task_state': u'shelving'}
2019-05-28 03:18:48.523 26 ERROR nova.api.openstack.wsgi Traceback (most
recent call last):
2019-05-28 03:18:48.523 26 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 788, in
wrapped
2019-05-28 03:18:48.523 26 ERROR nova.api.openstack.wsgi return f(*args,
**kwargs)
2019-05-28 03:18:48.523 26 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/shelve.py", line
43, in _shelve
2019-05-28 03:18:48.523 26 ERROR nova.api.openstack.wsgi
self.compute_api.shelve(context, instance)
2019-05-28 03:18:48.523 26 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 204, in inner
2019-05-28 03:18:48.523 26 ERROR nova.api.openstack.wsgi return
function(self, context, instance, *args, **kwargs)
2019-05-28
[Yahoo-eng-team] [Bug 1850682] [NEW] functional tests in rocky randomly fail with "Build of instance was re-scheduled: Cannot modify readonly field uuid"
Public bug reported: Seen here: https://561c74891cdc1994ef28-bf4df36ee9e72417a89c120068385812.ssl.cf2.rackcdn.com/690721/4/check /nova-tox-functional/5b7bec9/testr_results.html.gz >From logstash this is probably the earliest patch to hit it: https://review.opendev.org/#/c/687563/ http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22was %20re- scheduled%3A%20Cannot%20modify%20readonly%20field%20uuid%5C%22%20AND%20tags%3A%5C%22console%5C%22=7d Without something like this https://review.opendev.org/#/c/669545/ we can't tell where the original error is coming from though. ** Affects: nova Importance: Undecided Status: Confirmed ** Tags: compute gate-failure serviceability testing ** Changed in: nova Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1850682 Title: functional tests in rocky randomly fail with "Build of instance was re-scheduled: Cannot modify readonly field uuid" Status in OpenStack Compute (nova): Confirmed Bug description: Seen here: https://561c74891cdc1994ef28-bf4df36ee9e72417a89c120068385812.ssl.cf2.rackcdn.com/690721/4/check /nova-tox-functional/5b7bec9/testr_results.html.gz From logstash this is probably the earliest patch to hit it: https://review.opendev.org/#/c/687563/ http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22was %20re- scheduled%3A%20Cannot%20modify%20readonly%20field%20uuid%5C%22%20AND%20tags%3A%5C%22console%5C%22=7d Without something like this https://review.opendev.org/#/c/669545/ we can't tell where the original error is coming from though. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1850682/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1593155] Re: over_committed_disk_size is wrong for sparse flat files
*** This bug is a duplicate of bug 1764489 *** https://bugs.launchpad.net/bugs/1764489 ** This bug has been marked a duplicate of bug 1764489 Preallocated disks are deducted twice from disk_available_least when using preallocated_images = space -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1593155 Title: over_committed_disk_size is wrong for sparse flat files Status in OpenStack Compute (nova): In Progress Bug description: The libvirt driver creates flat disks as sparse by default. However, it always returns over_committed_disk_size=0 for flat disks in _get_instance_disk_info(). This incorrect data ends up being reported to the scheduler in the libvirt driver's get_available_resource() via _get_disk_over_committed_size_total(). _get_instance_disk_info() should use allocated blocks, not file size, when calculating over_commited_disk_size for flat disks. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1593155/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1663456] Re: Field 'updated_at' always 'None' when show aggregate
This could probably be considered the same as bug 1719561.
** Changed in: nova
Status: Opinion => Confirmed
** Changed in: nova
Importance: Wishlist => Low
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1663456
Title:
Field 'updated_at' always 'None' when show aggregate
Status in OpenStack Compute (nova):
Confirmed
Bug description:
Description
===
When i got detailed info of one host aggregate with CLI `openstack aggregate
show`,
the field 'updated_at' always was 'None'.
Steps to reproduce
==
* Create one host aggregate with CLI `openstack aggregate create t-sh`
* Set some properties for the aggregate with CLI `openstack aggregate set
--zone tztz --property foo=bar agg-sh`
* Get detailed info of the aggregate with CLI `openstack aggregate show
agg-sh`
Expected result
===
| updated_at| 2017-02-10T03:27:25.535045 |
Actual result
=
| updated_at| None |
Environment
===
1. nova version
[root@controller nova]# git log
commit 50d402821be7476eb58ccd791c50d8ed801e85eb
Author: Matt Riedemann
Date: Wed Feb 8 10:23:14 2017 -0500
Consider startup scenario in _get_compute_nodes_in_db
2. Which hypervisor did you use?
devstack + libvirt + kvm
Logs
==
Enable --debug in openstack command.
* Set some properties for the aggregate with '--debug'.
RESP BODY: {"aggregate": {"name": "agg-sh", "availability_zone": "tztz",
"deleted": false, "created_at": "2017-02-10T03:26:21.00", "updated_at":
"2017-02-10T03:27:25.535045", "hosts": [], "deleted_at": null, "id": 4,
"metadata": {"foo": "bar", "availability_zone": "tztz"}}}
Note: field 'updated_at' has valid value.
* Get detailed info with '--debug'
RESP BODY: {"aggregates": [{"name": "agg-1", "availability_zone": "tz1",
"deleted": false, "created_at": "2017-02-10T02:09:47.00", "updated_at":
null, "hosts": ["controller"], "deleted_at": null, "id": 1, "metadata":
{"color": "green", "foo": "bar", "availability_zone": "tz1"}}, {"name":
"agg-a", "availability_zone": "tz2", "deleted": false, "created_at":
"2017-02-10T02:39:15.00", "updated_at": null, "hosts": [], "deleted_at":
null, "id": 2, "metadata": {"foo": "tar", "availability_zone": "tz2"}},
{"name": "t-sh", "availability_zone": "tz3", "deleted": false, "created_at":
"2017-02-10T02:39:24.00", "updated_at": null, "hosts": [], "deleted_at":
null, "id": 3, "metadata": {"color": "blue", "hello": "world",
"availability_zone": "tz3"}}, {"name": "agg-sh", "availability_zone": "tztz",
"deleted": false, "created_at": "2017-02-10T03:26:21.00", "updated_at":
null, "hosts": [], "deleted_at": null, "id": 4, "metadata": {"foo": "bar",
"availability_zone": "tztz"}}]}
Note: field 'updated_at' is null.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1663456/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1663456] Re: Field 'updated_at' always 'None' when show aggregate
** Changed in: nova
Importance: Medium => Wishlist
** Changed in: nova
Status: In Progress => Opinion
** Changed in: nova
Assignee: Vishakha Agarwal (vishakha.agarwal) => (unassigned)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1663456
Title:
Field 'updated_at' always 'None' when show aggregate
Status in OpenStack Compute (nova):
Confirmed
Bug description:
Description
===
When i got detailed info of one host aggregate with CLI `openstack aggregate
show`,
the field 'updated_at' always was 'None'.
Steps to reproduce
==
* Create one host aggregate with CLI `openstack aggregate create t-sh`
* Set some properties for the aggregate with CLI `openstack aggregate set
--zone tztz --property foo=bar agg-sh`
* Get detailed info of the aggregate with CLI `openstack aggregate show
agg-sh`
Expected result
===
| updated_at| 2017-02-10T03:27:25.535045 |
Actual result
=
| updated_at| None |
Environment
===
1. nova version
[root@controller nova]# git log
commit 50d402821be7476eb58ccd791c50d8ed801e85eb
Author: Matt Riedemann
Date: Wed Feb 8 10:23:14 2017 -0500
Consider startup scenario in _get_compute_nodes_in_db
2. Which hypervisor did you use?
devstack + libvirt + kvm
Logs
==
Enable --debug in openstack command.
* Set some properties for the aggregate with '--debug'.
RESP BODY: {"aggregate": {"name": "agg-sh", "availability_zone": "tztz",
"deleted": false, "created_at": "2017-02-10T03:26:21.00", "updated_at":
"2017-02-10T03:27:25.535045", "hosts": [], "deleted_at": null, "id": 4,
"metadata": {"foo": "bar", "availability_zone": "tztz"}}}
Note: field 'updated_at' has valid value.
* Get detailed info with '--debug'
RESP BODY: {"aggregates": [{"name": "agg-1", "availability_zone": "tz1",
"deleted": false, "created_at": "2017-02-10T02:09:47.00", "updated_at":
null, "hosts": ["controller"], "deleted_at": null, "id": 1, "metadata":
{"color": "green", "foo": "bar", "availability_zone": "tz1"}}, {"name":
"agg-a", "availability_zone": "tz2", "deleted": false, "created_at":
"2017-02-10T02:39:15.00", "updated_at": null, "hosts": [], "deleted_at":
null, "id": 2, "metadata": {"foo": "tar", "availability_zone": "tz2"}},
{"name": "t-sh", "availability_zone": "tz3", "deleted": false, "created_at":
"2017-02-10T02:39:24.00", "updated_at": null, "hosts": [], "deleted_at":
null, "id": 3, "metadata": {"color": "blue", "hello": "world",
"availability_zone": "tz3"}}, {"name": "agg-sh", "availability_zone": "tztz",
"deleted": false, "created_at": "2017-02-10T03:26:21.00", "updated_at":
null, "hosts": [], "deleted_at": null, "id": 4, "metadata": {"foo": "bar",
"availability_zone": "tztz"}}]}
Note: field 'updated_at' is null.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1663456/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1850514] Re: ReshapeForPCPUsTest.test_vcpu_to_pcpu_reshape intermittently fails with "Cannot 'migrate' instance while it is in vm_state building"
** Also affects: nova/train
Importance: Undecided
Status: New
** Changed in: nova/train
Status: New => Confirmed
** Changed in: nova/train
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1850514
Title:
ReshapeForPCPUsTest.test_vcpu_to_pcpu_reshape intermittently fails
with "Cannot 'migrate' instance while it is in vm_state building"
Status in OpenStack Compute (nova):
In Progress
Status in OpenStack Compute (nova) train series:
Confirmed
Bug description:
Seen here:
https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f05/691390/6/check
/nova-tox-functional/f059dc0/testr_results.html.gz
ft1.1:
nova.tests.functional.libvirt.test_numa_servers.ReshapeForPCPUsTest.test_vcpu_to_pcpu_reshapetesttools.testresult.real._StringException:
pythonlogging:'': {{{
2019-10-29 17:33:28,372 WARNING [placement.db_api] TransactionFactory already
started, not reconfiguring.
2019-10-29 17:33:28,423 INFO [nova.service] Starting conductor node (version
20.1.0)
2019-10-29 17:33:28,478 INFO [nova.service] Starting scheduler node (version
20.1.0)
2019-10-29 17:33:29,023 INFO [nova.virt.driver] Loading compute driver
'libvirt.LibvirtDriver'
2019-10-29 17:33:29,026 WARNING [os_brick.initiator.connectors.remotefs]
Connection details not present. RemoteFsClient may not initialize properly.
2019-10-29 17:33:29,028 INFO [nova.service] Starting compute node (version
20.1.0)
2019-10-29 17:33:29,030 WARNING [nova.virt.libvirt.driver] The 'vcpu_pin_set'
config option has been deprecated and will be removed in a future release. When
defined, 'vcpu_pin_set' will be used to calculate 'VCPU' inventory and schedule
instances that have 'VCPU' allocations. If you wish to define specific host
CPUs to be used for 'VCPU' or 'PCPU' inventory, you must migrate the
'vcpu_pin_set' config option value to '[compute] cpu_shared_set' and '[compute]
cpu_dedicated_set', respectively, and undefine 'vcpu_pin_set'.
2019-10-29 17:33:29,038 WARNING [nova.virt.libvirt.driver] my_ip address
(38.108.68.36) was not found on any of the interfaces:
2019-10-29 17:33:29,039 WARNING [nova.virt.libvirt.driver] Running Nova with
a libvirt version less than 4.0.0 is deprecated. The required minimum version
of libvirt will be raised to 4.0.0 in the next release.
2019-10-29 17:33:29,039 WARNING [nova.virt.libvirt.driver] Running Nova with
a QEMU version less than 2.11.0 is deprecated. The required minimum version of
QEMU will be raised to 2.11.0 in the next release.
2019-10-29 17:33:29,059 WARNING [nova.compute.manager] Compute node
test_compute0 not found in the database. If this is the first time this service
is starting on this host, then you can ignore this warning.
2019-10-29 17:33:29,066 INFO [nova.compute.manager] Looking for unclaimed
instances stuck in BUILDING status for nodes managed by this host
2019-10-29 17:33:29,081 WARNING [nova.compute.manager] No compute node record
found for host test_compute0. If this is the first time this service is
starting on this host, then you can ignore this warning.
2019-10-29 17:33:29,094 WARNING [nova.compute.resource_tracker] No compute
node record for test_compute0:test_compute0
2019-10-29 17:33:29,101 INFO [nova.compute.resource_tracker] Compute node
record created for test_compute0:test_compute0 with uuid:
cada8f90-3f3d-4f22-8312-770a0a818828
2019-10-29 17:33:29,179 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/resource_providers?in_tree=cada8f90-3f3d-4f22-8312-770a0a818828"
status: 200 len: 26 microversion: 1.14
2019-10-29 17:33:29,192 INFO [placement.requestlog] 127.0.0.1 "POST
/placement/resource_providers" status: 200 len: 836 microversion: 1.20
2019-10-29 17:33:29,193 INFO [nova.scheduler.client.report]
[req-3fabe3ee-b8f4-4d5e-9c62-185c0ae18c74] Created resource provider record via
placement API for resource provider with UUID
cada8f90-3f3d-4f22-8312-770a0a818828 and name test_compute0.
2019-10-29 17:33:29,195 INFO [nova.virt.libvirt.host] kernel doesn't support
AMD SEV
2019-10-29 17:33:29,227 INFO [placement.requestlog] 127.0.0.1 "PUT
/placement/resource_providers/cada8f90-3f3d-4f22-8312-770a0a818828/inventories"
status: 200 len: 405 microversion: 1.26
2019-10-29 17:33:29,241 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/traits?name=in:COMPUTE_IMAGE_TYPE_ARI,COMPUTE_IMAGE_TYPE_AKI,COMPUTE_VOLUME_EXTEND,COMPUTE_IMAGE_TYPE_AMI,COMPUTE_IMAGE_TYPE_RAW,HW_CPU_X86_VMX,COMPUTE_NET_ATTACH_INTERFACE,COMPUTE_VOLUME_MULTI_ATTACH,HW_CPU_X86_AESNI,HW_CPU_HYPERTHREADING,COMPUTE_IMAGE_TYPE_QCOW2,COMPUTE_VOLUME_ATTACH_WITH_TAG,COMPUTE_NET_ATTACH_INTERFACE_WITH_TAG,COMPUTE_IMAGE_TYPE_ISO,COMPUTE_TRUSTED_CERTS,COMPUTE_DEVICE_TAGGING"
status: 200 len: 447 microversion: 1.6
2019-10-29 17:33:29,266 INFO
[Yahoo-eng-team] [Bug 1850514] [NEW] ReshapeForPCPUsTest.test_vcpu_to_pcpu_reshape intermittently fails with "Cannot 'migrate' instance while it is in vm_state building"
Public bug reported:
Seen here:
https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f05/691390/6/check
/nova-tox-functional/f059dc0/testr_results.html.gz
ft1.1:
nova.tests.functional.libvirt.test_numa_servers.ReshapeForPCPUsTest.test_vcpu_to_pcpu_reshapetesttools.testresult.real._StringException:
pythonlogging:'': {{{
2019-10-29 17:33:28,372 WARNING [placement.db_api] TransactionFactory already
started, not reconfiguring.
2019-10-29 17:33:28,423 INFO [nova.service] Starting conductor node (version
20.1.0)
2019-10-29 17:33:28,478 INFO [nova.service] Starting scheduler node (version
20.1.0)
2019-10-29 17:33:29,023 INFO [nova.virt.driver] Loading compute driver
'libvirt.LibvirtDriver'
2019-10-29 17:33:29,026 WARNING [os_brick.initiator.connectors.remotefs]
Connection details not present. RemoteFsClient may not initialize properly.
2019-10-29 17:33:29,028 INFO [nova.service] Starting compute node (version
20.1.0)
2019-10-29 17:33:29,030 WARNING [nova.virt.libvirt.driver] The 'vcpu_pin_set'
config option has been deprecated and will be removed in a future release. When
defined, 'vcpu_pin_set' will be used to calculate 'VCPU' inventory and schedule
instances that have 'VCPU' allocations. If you wish to define specific host
CPUs to be used for 'VCPU' or 'PCPU' inventory, you must migrate the
'vcpu_pin_set' config option value to '[compute] cpu_shared_set' and '[compute]
cpu_dedicated_set', respectively, and undefine 'vcpu_pin_set'.
2019-10-29 17:33:29,038 WARNING [nova.virt.libvirt.driver] my_ip address
(38.108.68.36) was not found on any of the interfaces:
2019-10-29 17:33:29,039 WARNING [nova.virt.libvirt.driver] Running Nova with a
libvirt version less than 4.0.0 is deprecated. The required minimum version of
libvirt will be raised to 4.0.0 in the next release.
2019-10-29 17:33:29,039 WARNING [nova.virt.libvirt.driver] Running Nova with a
QEMU version less than 2.11.0 is deprecated. The required minimum version of
QEMU will be raised to 2.11.0 in the next release.
2019-10-29 17:33:29,059 WARNING [nova.compute.manager] Compute node
test_compute0 not found in the database. If this is the first time this service
is starting on this host, then you can ignore this warning.
2019-10-29 17:33:29,066 INFO [nova.compute.manager] Looking for unclaimed
instances stuck in BUILDING status for nodes managed by this host
2019-10-29 17:33:29,081 WARNING [nova.compute.manager] No compute node record
found for host test_compute0. If this is the first time this service is
starting on this host, then you can ignore this warning.
2019-10-29 17:33:29,094 WARNING [nova.compute.resource_tracker] No compute node
record for test_compute0:test_compute0
2019-10-29 17:33:29,101 INFO [nova.compute.resource_tracker] Compute node
record created for test_compute0:test_compute0 with uuid:
cada8f90-3f3d-4f22-8312-770a0a818828
2019-10-29 17:33:29,179 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/resource_providers?in_tree=cada8f90-3f3d-4f22-8312-770a0a818828"
status: 200 len: 26 microversion: 1.14
2019-10-29 17:33:29,192 INFO [placement.requestlog] 127.0.0.1 "POST
/placement/resource_providers" status: 200 len: 836 microversion: 1.20
2019-10-29 17:33:29,193 INFO [nova.scheduler.client.report]
[req-3fabe3ee-b8f4-4d5e-9c62-185c0ae18c74] Created resource provider record via
placement API for resource provider with UUID
cada8f90-3f3d-4f22-8312-770a0a818828 and name test_compute0.
2019-10-29 17:33:29,195 INFO [nova.virt.libvirt.host] kernel doesn't support
AMD SEV
2019-10-29 17:33:29,227 INFO [placement.requestlog] 127.0.0.1 "PUT
/placement/resource_providers/cada8f90-3f3d-4f22-8312-770a0a818828/inventories"
status: 200 len: 405 microversion: 1.26
2019-10-29 17:33:29,241 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/traits?name=in:COMPUTE_IMAGE_TYPE_ARI,COMPUTE_IMAGE_TYPE_AKI,COMPUTE_VOLUME_EXTEND,COMPUTE_IMAGE_TYPE_AMI,COMPUTE_IMAGE_TYPE_RAW,HW_CPU_X86_VMX,COMPUTE_NET_ATTACH_INTERFACE,COMPUTE_VOLUME_MULTI_ATTACH,HW_CPU_X86_AESNI,HW_CPU_HYPERTHREADING,COMPUTE_IMAGE_TYPE_QCOW2,COMPUTE_VOLUME_ATTACH_WITH_TAG,COMPUTE_NET_ATTACH_INTERFACE_WITH_TAG,COMPUTE_IMAGE_TYPE_ISO,COMPUTE_TRUSTED_CERTS,COMPUTE_DEVICE_TAGGING"
status: 200 len: 447 microversion: 1.6
2019-10-29 17:33:29,266 INFO [placement.requestlog] 127.0.0.1 "PUT
/placement/resource_providers/cada8f90-3f3d-4f22-8312-770a0a818828/traits"
status: 200 len: 482 microversion: 1.6
2019-10-29 17:33:29,284 INFO [placement.requestlog] 127.0.0.1 "GET
/placement/resource_providers?name=test_compute0" status: 200 len: 440
microversion: 1.0
2019-10-29 17:33:29,328 INFO [nova.virt.driver] Loading compute driver
'libvirt.LibvirtDriver'
2019-10-29 17:33:29,332 WARNING [os_brick.initiator.connectors.remotefs]
Connection details not present. RemoteFsClient may not initialize properly.
2019-10-29 17:33:29,334 INFO [nova.service] Starting compute node (version
20.1.0)
2019-10-29 17:33:29,336 WARNING
[Yahoo-eng-team] [Bug 1845600] Re: Install and configure controller node (nova) missing packet
The nova-console service is xen driver specific and removed on purpose because it's deprecated, see: https://review.opendev.org/#/c/610075/ ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1845600 Title: Install and configure controller node (nova) missing packet Status in OpenStack Compute (nova): Invalid Bug description: This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes: - [x] This doc is inaccurate in this way: # in section "Install & configure components" -> chapter 1 the packet "openstack-nova-console" is missing. I am installing Openstack Stein on CentOS 7 - [ ] This is a doc addition request. - [x] I have a fix to the document that I can paste below including example: input and output. Input: # yum install openstack-nova-api openstack-nova-conductor \ openstack-nova-novncproxy openstack-nova-scheduler output: # yum install openstack-nova-api openstack-nova-conductor \ openstack-nova-novncproxy openstack-nova-scheduler \ openstack-nova-console If you have a troubleshooting or support issue, use the following resources: - Ask OpenStack: http://ask.openstack.org - The mailing list: http://lists.openstack.org - IRC: 'openstack' channel on Freenode --- Release: on 2019-03-27 10:24:01 SHA: b54c11dedad79ec18ccd6009c67a1d8a94627dde Source: https://git.openstack.org/cgit/openstack/nova/tree/doc/source/install/controller-install-rdo.rst URL: https://docs.openstack.org/nova/stein/install/controller-install-rdo.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1845600/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1475112] Re: Instance should return to its initial state instead of ACTIVE when resize fails
*** This bug is a duplicate of bug 1811235 ***
https://bugs.launchpad.net/bugs/1811235
** This bug is no longer a duplicate of bug 1551703
Resize a vm that vm_state is "stopped" failure， vm's task_state rollback to
"active"
** This bug has been marked a duplicate of bug 1811235
instance's vm-state becomes error when cold-migrate instance to same host
failed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1475112
Title:
Instance should return to its initial state instead of ACTIVE when
resize fails
Status in OpenStack Compute (nova):
In Progress
Bug description:
When instance resize fails, instead of return to its inital vm_state,
it is set to ACTIVE:
Creating an instance with flavor 2:
root@kevin-dev:/opt/stack# nova boot --flavor 2 --nic
net-id=13854918-10d5-466e-b576-1c4c9689f357 --image
5e992b08-d772-44d7-90bf-0b0d9eaa2fcd test1
+--++
| Property | Value
|
+--++
| OS-DCF:diskConfig| MANUAL
|
| OS-EXT-AZ:availability_zone | nova
|
| OS-EXT-SRV-ATTR:host | -
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | -
|
| OS-EXT-SRV-ATTR:instance_name| instance-003a
|
| OS-EXT-STS:power_state | 0
|
| OS-EXT-STS:task_state| scheduling
|
| OS-EXT-STS:vm_state | building
|
| OS-SRV-USG:launched_at | -
|
| OS-SRV-USG:terminated_at | -
|
| accessIPv4 |
|
| accessIPv6 |
|
| adminPass| wFUucx56wSEZ
|
| config_drive |
|
| created | 2015-07-16T03:19:45Z
|
| flavor | m1.small (2)
|
| hostId |
|
| id | 0e74eff9-7a25-4a9d-b894-cf4db80dc1c4
|
| image| cirros-0.3.4-x86_64-uec
(5e992b08-d772-44d7-90bf-0b0d9eaa2fcd) |
| key_name | -
|
| metadata | {}
|
| name | test1
|
| os-extended-volumes:volumes_attached | []
|
| progress | 0
|
| security_groups | default
|
| status | BUILD
|
| tenant_id| 939300520ec34fea975ef197c8fd61b8
|
| updated | 2015-07-16T03:19:45Z
|
| user_id | 44c538801d8f46fa9b9b15d7d7986bef
|
+--++
root@kevin-dev:/opt/stack# nova list
+--+---+++-+-+
| ID | Name | Status | Task State | Power
State | Networks|
[Yahoo-eng-team] [Bug 1850437] Re: changePassword action allows adminPass="" value
This is wrong, it looks like empty string was always allowed, going back to this code: https://github.com/openstack/nova/blob/icehouse- eol/nova/api/openstack/compute/servers.py#L1490 ** Changed in: nova Status: In Progress => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1850437 Title: changePassword action allows adminPass="" value Status in OpenStack Compute (nova): Invalid Bug description: The changePassword API requires an adminPass value: https://docs.openstack.org/api-ref/compute/?expanded=change- administrative-password-changepassword-action-detail#change- administrative-password-changepassword-action But the schema allows an empty string: https://github.com/openstack/nova/blob/9742a64403c0a0ae5e0b37df5b0bf3ba14ac4626/nova/api/openstack/compute/schemas/admin_password.py#L24 https://github.com/openstack/nova/blob/9742a64403c0a0ae5e0b37df5b0bf3ba14ac4626/nova/api/validation/parameter_types.py#L370 This is evident from the unit test here that doesn't fail: https://github.com/openstack/nova/blob/9742a64403c0a0ae5e0b37df5b0bf3ba14ac4626/nova/tests/unit/api/openstack/compute/test_admin_password.py#L61 Looking at old changes like this: https://review.opendev.org/#/c/145398/ It looks like the legacy v2 API did not allow empty string or None values for adminPass but that was regressed with the schema validation added to the changePassword API here: https://review.opendev.org/#/c/59598/ And we can see from https://review.opendev.org/#/c/35625/ that an adminPass="" would not have been supported originally but was regressed in the schema change since there was no test for empty string at that the time of the schema addition. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1850437/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1850437] [NEW] changePassword action allows adminPass="" value
Public bug reported: The changePassword API requires an adminPass value: https://docs.openstack.org/api-ref/compute/?expanded=change- administrative-password-changepassword-action-detail#change- administrative-password-changepassword-action But the schema allows an empty string: https://github.com/openstack/nova/blob/9742a64403c0a0ae5e0b37df5b0bf3ba14ac4626/nova/api/openstack/compute/schemas/admin_password.py#L24 https://github.com/openstack/nova/blob/9742a64403c0a0ae5e0b37df5b0bf3ba14ac4626/nova/api/validation/parameter_types.py#L370 This is evident from the unit test here that doesn't fail: https://github.com/openstack/nova/blob/9742a64403c0a0ae5e0b37df5b0bf3ba14ac4626/nova/tests/unit/api/openstack/compute/test_admin_password.py#L61 Looking at old changes like this: https://review.opendev.org/#/c/145398/ It looks like the legacy v2 API did not allow empty string or None values for adminPass but that was regressed with the schema validation added to the changePassword API here: https://review.opendev.org/#/c/59598/ And we can see from https://review.opendev.org/#/c/35625/ that an adminPass="" would not have been supported originally but was regressed in the schema change since there was no test for empty string at that the time of the schema addition. ** Affects: nova Importance: Low Status: Triaged ** Tags: api ** Changed in: nova Status: New => Triaged ** Changed in: nova Importance: Undecided => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1850437 Title: changePassword action allows adminPass="" value Status in OpenStack Compute (nova): Triaged Bug description: The changePassword API requires an adminPass value: https://docs.openstack.org/api-ref/compute/?expanded=change- administrative-password-changepassword-action-detail#change- administrative-password-changepassword-action But the schema allows an empty string: https://github.com/openstack/nova/blob/9742a64403c0a0ae5e0b37df5b0bf3ba14ac4626/nova/api/openstack/compute/schemas/admin_password.py#L24 https://github.com/openstack/nova/blob/9742a64403c0a0ae5e0b37df5b0bf3ba14ac4626/nova/api/validation/parameter_types.py#L370 This is evident from the unit test here that doesn't fail: https://github.com/openstack/nova/blob/9742a64403c0a0ae5e0b37df5b0bf3ba14ac4626/nova/tests/unit/api/openstack/compute/test_admin_password.py#L61 Looking at old changes like this: https://review.opendev.org/#/c/145398/ It looks like the legacy v2 API did not allow empty string or None values for adminPass but that was regressed with the schema validation added to the changePassword API here: https://review.opendev.org/#/c/59598/ And we can see from https://review.opendev.org/#/c/35625/ that an adminPass="" would not have been supported originally but was regressed in the schema change since there was no test for empty string at that the time of the schema addition. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1850437/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1850412] [NEW] Useful image properties in glance - os_admin_user not documented
Public bug reported: - [x] This is a doc addition request. The os_admin_user image property was added to nova in the liberty release: https://specs.openstack.org/openstack/nova- specs/specs/liberty/implemented/libvirt-set-admin-password.html It's used to specify the admin username in a guest, otherwise nova defaults to root for linux guests and Administrator for windows guests. It's used when changing the admin password in the guest. There is a metadef for the property so we could just use that for the docs: https://github.com/openstack/glance/blob/54329c6a21b0d3f845b09e79f710fc795976a175/etc/metadefs /operating-system.json#L26 Only the libvirt driver in nova currently uses the os_admin_user image property. --- Release: on 2019-09-12 12:02:50 SHA: 99938b8afeebcdafc7baa07b73b7762abcc646a7 Source: https://opendev.org/openstack/glance/src/doc/source/admin/useful-image-properties.rst URL: https://docs.openstack.org/glance/latest/admin/useful-image-properties.html ** Affects: glance Importance: Undecided Status: New ** Tags: documentation -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1850412 Title: Useful image properties in glance - os_admin_user not documented Status in Glance: New Bug description: - [x] This is a doc addition request. The os_admin_user image property was added to nova in the liberty release: https://specs.openstack.org/openstack/nova- specs/specs/liberty/implemented/libvirt-set-admin-password.html It's used to specify the admin username in a guest, otherwise nova defaults to root for linux guests and Administrator for windows guests. It's used when changing the admin password in the guest. There is a metadef for the property so we could just use that for the docs: https://github.com/openstack/glance/blob/54329c6a21b0d3f845b09e79f710fc795976a175/etc/metadefs /operating-system.json#L26 Only the libvirt driver in nova currently uses the os_admin_user image property. --- Release: on 2019-09-12 12:02:50 SHA: 99938b8afeebcdafc7baa07b73b7762abcc646a7 Source: https://opendev.org/openstack/glance/src/doc/source/admin/useful-image-properties.rst URL: https://docs.openstack.org/glance/latest/admin/useful-image-properties.html To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1850412/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1849870] [NEW] docs build failing on psycopg2 dependency
Public bug reported:
https://review.opendev.org/#/c/689152/ re-introduced using the
requirements.txt file during docs builds which it shouldn't be because
it installs things like psycopg2 which we don't need and that's failing
now:
https://zuul.opendev.org/t/openstack/build/10344924bca44849b06b8fae29b5494f/log
/job-output.txt#652
2019-10-25 13:58:32.599585 | ubuntu-bionic | Downloading
http://mirror.gra1.ovh.openstack.org/pypifiles/packages/12/67/5d953cb20497d4f56965bc5bcf03134244be7bae4eb2b1f7ca5cf31b245f/os_xenapi-0.3.4-py2.py3
-none-any.whl (137kB)
2019-10-25 13:58:32.599697 | ubuntu-bionic | Collecting psycopg2===2.8.4
(from -c /home/zuul/src/opendev.org/openstack/requirements/upper-
constraints.txt (line 97))
2019-10-25 13:58:32.599851 | ubuntu-bionic | Downloading
http://mirror.gra1.ovh.openstack.org/pypifiles/packages/84/d7/6a93c99b5ba4d4d22daa3928b983cec66df4536ca50b22ce5dcac65e4e71/psycopg2-2.8.4.tar.gz
(377kB)
2019-10-25 13:58:32.599905 | ubuntu-bionic | ERROR: Command errored
out with exit status 1:
2019-10-25 13:58:32.600316 | ubuntu-bionic | command:
/home/zuul/src/opendev.org/openstack/nova/.tox/docs/bin/python -c
'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-
ki5_j1sg/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-
ki5_j1sg/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"',
open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
egg_info --egg-base pip-egg-info
2019-10-25 13:58:32.600384 | ubuntu-bionic | cwd: /tmp/pip-
install-ki5_j1sg/psycopg2/
2019-10-25 13:58:32.600421 | ubuntu-bionic | Complete output (7
lines):
2019-10-25 13:58:32.600449 | ubuntu-bionic | running egg_info
2019-10-25 13:58:32.600496 | ubuntu-bionic | creating pip-egg-
info/psycopg2.egg-info
2019-10-25 13:58:32.600551 | ubuntu-bionic | writing pip-egg-
info/psycopg2.egg-info/PKG-INFO
2019-10-25 13:58:32.600631 | ubuntu-bionic | writing
dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
2019-10-25 13:58:32.600706 | ubuntu-bionic | writing top-level names
to pip-egg-info/psycopg2.egg-info/top_level.txt
2019-10-25 13:58:32.600780 | ubuntu-bionic | writing manifest file
'pip-egg-info/psycopg2.egg-info/SOURCES.txt'
2019-10-25 13:58:32.600936 | ubuntu-bionic | Error: b'You need to
install postgresql-server-dev-X.Y for building a server-side extension
or libpq-dev for building a client-side application.\n'
** Affects: nova
Importance: Critical
Assignee: Matt Riedemann (mriedem)
Status: Confirmed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1849870
Title:
docs build failing on psycopg2 dependency
Status in OpenStack Compute (nova):
Confirmed
Bug description:
https://review.opendev.org/#/c/689152/ re-introduced using the
requirements.txt file during docs builds which it shouldn't be because
it installs things like psycopg2 which we don't need and that's
failing now:
https://zuul.opendev.org/t/openstack/build/10344924bca44849b06b8fae29b5494f/log
/job-output.txt#652
2019-10-25 13:58:32.599585 | ubuntu-bionic | Downloading
http://mirror.gra1.ovh.openstack.org/pypifiles/packages/12/67/5d953cb20497d4f56965bc5bcf03134244be7bae4eb2b1f7ca5cf31b245f/os_xenapi-0.3.4-py2.py3
-none-any.whl (137kB)
2019-10-25 13:58:32.599697 | ubuntu-bionic | Collecting
psycopg2===2.8.4 (from -c
/home/zuul/src/opendev.org/openstack/requirements/upper-
constraints.txt (line 97))
2019-10-25 13:58:32.599851 | ubuntu-bionic | Downloading
http://mirror.gra1.ovh.openstack.org/pypifiles/packages/84/d7/6a93c99b5ba4d4d22daa3928b983cec66df4536ca50b22ce5dcac65e4e71/psycopg2-2.8.4.tar.gz
(377kB)
2019-10-25 13:58:32.599905 | ubuntu-bionic | ERROR: Command
errored out with exit status 1:
2019-10-25 13:58:32.600316 | ubuntu-bionic | command:
/home/zuul/src/opendev.org/openstack/nova/.tox/docs/bin/python -c
'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-
install-ki5_j1sg/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-
install-ki5_j1sg/psycopg2/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
egg_info --egg-base pip-egg-info
2019-10-25 13:58:32.600384 | ubuntu-bionic | cwd: /tmp/pip-
install-ki5_j1sg/psycopg2/
2019-10-25 13:58:32.600421 | ubuntu-bionic | Complete output (7
lines):
2019-10-25 13:58:32.600449 | ubuntu-bionic | running egg_info
[Yahoo-eng-team] [Bug 1849850] [NEW] F841 is not checked since Train
Public bug reported: With this change in Train: https://review.opendev.org/#/c/651553/ We're using pycodestyle rather than pep8 and pycodestyle doesn't check for F841: https://pycodestyle.readthedocs.io/en/latest/intro.html#error-codes Which means things like this slip through and then fail on backports to https://review.opendev.org/#/c/691282/ It would be good to see if we can get F841 checked on master and train again so we don't have that difference with stable branches. I'm not sure how trivial that is though, like if we can separately run flake8 with F841 during our pep8 runs. This is just to track the issue but I realize it might be closed as Won't Fix. ** Affects: nova Importance: Undecided Status: Triaged ** Tags: pep8 testing -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1849850 Title: F841 is not checked since Train Status in OpenStack Compute (nova): Triaged Bug description: With this change in Train: https://review.opendev.org/#/c/651553/ We're using pycodestyle rather than pep8 and pycodestyle doesn't check for F841: https://pycodestyle.readthedocs.io/en/latest/intro.html#error-codes Which means things like this slip through and then fail on backports to https://review.opendev.org/#/c/691282/ It would be good to see if we can get F841 checked on master and train again so we don't have that difference with stable branches. I'm not sure how trivial that is though, like if we can separately run flake8 with F841 during our pep8 runs. This is just to track the issue but I realize it might be closed as Won't Fix. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1849850/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1849409] Re: openstack server list --deleted --limit -1 hangs
** Also affects: nova/rocky Importance: Undecided Status: New ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/ocata Importance: Undecided Status: New ** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/pike Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1849409 Title: openstack server list --deleted --limit -1 hangs Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) ocata series: New Status in OpenStack Compute (nova) pike series: New Status in OpenStack Compute (nova) queens series: New Status in OpenStack Compute (nova) rocky series: New Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: New Bug description: OpenStack Rocky: When running `openstack server list --deleted --limit -1 hangs` it will hang and not return. A debug output is found in this pastebin http://paste.openstack.org/show/785497/ and a direct curl is shown here http://paste.openstack.org/show/785488/ It seems to be related to the marker as discussed here. http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack- nova.2019-10-22.log.html#t2019-10-22T20:35:11 after the initial collection of 18 servers, for some reason it than tried to grab with https://192.168.23.35:8774/v2.1/428982d4248a419a933668b6a4dd14a0/servers/dtail?deleted=True=8aab9854 -af8e-4e98-840a-192f15ae01f9 where the marker is the 18th deleted server in the list. it then loops on this until you breakout. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1849409/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1849481] Re: Failed to delete volumes while delete instance
>From the logs I see cinder.auth_type = None so the [cinder] section of the nova configuration is not configured for a service user to act on behalf of the user to detach and delete the volume. With reclaim_instance_interval>0 the periodic task in the compute service will delete the server and the volume but doesn't have the user token so you must configure nova to have admin rights to work with cinder on the user's behalf. For example: https://review.opendev.org/#/c/685488/ I'm going to mark this as invalid since it's a misconfiguration, though arguably we should update the documentation for the reclaim_instance_interval option to mention needing to configure cinder to properly clean up volumes when the server is deleted. ** Changed in: nova Status: New => Invalid ** Changed in: nova Status: Invalid => Triaged -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1849481 Title: Failed to delete volumes while delete instance Status in OpenStack Compute (nova): Triaged Bug description: Brief Description - To launch an instance and select "delete volume while deleting instance". After the instance deleted, the attached volume still remained. please refer to attached file as nova_compute.log Steps to Reproduce -- 1. modify "reclaim_instance_interval" value from 0 -> 180 in nova/conf/compute.py on docker nova_api and nova compute 2. docker restart nova_api & nova_compute 3. login to Openstack from browser, and launch an instance with "delete volume while deleting instance" 4. Delete instance 5. Check attached volume still be there, and status is "running" Expected Behavior -- Delete instance and attached volume should be deleted also. Enviroment -- Broswer: Chrome 77.0.3865.120 OS: macOS Mojave 10.14.6 Openstack version: stein Openstack deployed system: Centos Linux version 3.10.0-1062.1.2.el7.x86_64 Reproducibility --- 100% reproduce able. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1849481/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1554440] Re: Show volume attachment exception does not looks good if server is in shelved_offloaded state
There was disagreement in the patch on whether or not a microversion was necessary to fix this (I don't think it is but that's just my opinion here). We could have fixed this in the general cleanup 2.75 microversion but forgot about it, so I'm just going to mark this as won't fix since it doesn't seem to be bothering anybody. ** Changed in: nova Status: In Progress => Won't Fix ** Changed in: nova Assignee: Matt Riedemann (mriedem) => (unassigned) ** Changed in: nova Importance: Medium => Wishlist -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1554440 Title: Show volume attachment exception does not looks good if server is in shelved_offloaded state Status in OpenStack Compute (nova): Fix Released Bug description: In microversion v2.20, Nova allows to attach/detach volume on server in shelved or shelved_offloaded state. When server is in shelved_offloaded state, it means server is not on any host. And volume attach operation is done directly via cinder. In that case device volume mount will be defered until the instance is unshelved. Which is valid case and all fine till here. Now issue is if users does the list attachments for server in shelved_offloaded state, it return the attachements without 'device' But Show volume attachment throw exception as there is no mount point for that attachment - https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/volumes.py#L289 List and show attachments should be consistent for server in shelved_offloaded state. Although error message if volume not mount - "volume_id not found: %s" does not sounds very nice here for other case too. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1554440/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1637483] Re: Add healthcheck middleware to pipelines
** Changed in: nova Assignee: Jesse Keating (jesse-keating) => (unassigned) ** Changed in: nova Importance: Undecided => Wishlist ** Changed in: nova Status: In Progress => Opinion -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1637483 Title: Add healthcheck middleware to pipelines Status in OpenStack Compute (nova): Opinion Bug description: As an operator, I want to be able to use oslo healthcheck middleware[1] to Nova's pipeline so that I can GET the /healthcheck URI to determine "up" or not for a given nova-api host. The healthcheck middleware allows for manually setting the health status to offline, without actually stopping the API service. When I can do this, I can easily "take an API node offline" from the aspect of the load balancer, which uses the healthcheck path for status checks, before stopping the API process itself. This is a quick and generic way to prevent new connections to a given API node while restarting it as part of a rolling restart. This middleware has already been added to glance[2], and I've got an open review to add it to keystone as well[3]. My eventual goal is to have it in use across all the services that directly listen for client connections. 1: http://docs.openstack.org/developer/oslo.middleware/healthcheck_plugins.html 2: https://review.openstack.org/#/c/148595/ 3: https://review.openstack.org/#/c/387731/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1637483/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1849351] [NEW] horizon doesn't work with python-novaclient 16.0.0
Public bug reported:
Seen here for the requirements bump change
https://review.opendev.org/#/c/690097/:
https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2e3/690097/1/check
/cross-horizon-py36/2e36786/job-output.txt
2019-10-22 15:06:33.720990 | ubuntu-bionic | Traceback (most recent call last):
2019-10-22 15:06:33.721129 | ubuntu-bionic | File "./manage.py", line 23, in
2019-10-22 15:06:33.721211 | ubuntu-bionic |
execute_from_command_line(sys.argv)
2019-10-22 15:06:33.721457 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/management/__init__.py",
line 371, in execute_from_command_line
2019-10-22 15:06:33.721513 | ubuntu-bionic | utility.execute()
2019-10-22 15:06:33.721733 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/management/__init__.py",
line 365, in execute
2019-10-22 15:06:33.721855 | ubuntu-bionic |
self.fetch_command(subcommand).run_from_argv(self.argv)
2019-10-22 15:06:33.722093 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/management/commands/test.py",
line 26, in run_from_argv
2019-10-22 15:06:33.722163 | ubuntu-bionic | super().run_from_argv(argv)
2019-10-22 15:06:33.722385 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/management/base.py",
line 288, in run_from_argv
2019-10-22 15:06:33.722463 | ubuntu-bionic | self.execute(*args,
**cmd_options)
2019-10-22 15:06:33.722678 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/management/base.py",
line 335, in execute
2019-10-22 15:06:33.722759 | ubuntu-bionic | output = self.handle(*args,
**options)
2019-10-22 15:06:33.722983 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/management/commands/test.py",
line 59, in handle
2019-10-22 15:06:33.723074 | ubuntu-bionic | failures =
test_runner.run_tests(test_labels)
2019-10-22 15:06:33.723279 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/test/runner.py",
line 602, in run_tests
2019-10-22 15:06:33.72 | ubuntu-bionic | self.run_checks()
2019-10-22 15:06:33.723539 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/test/runner.py",
line 564, in run_checks
2019-10-22 15:06:33.723633 | ubuntu-bionic | call_command('check',
verbosity=self.verbosity)
2019-10-22 15:06:33.723888 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/management/__init__.py",
line 141, in call_command
2019-10-22 15:06:33.723981 | ubuntu-bionic | return command.execute(*args,
**defaults)
2019-10-22 15:06:33.724196 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/management/base.py",
line 335, in execute
2019-10-22 15:06:33.724278 | ubuntu-bionic | output = self.handle(*args,
**options)
2019-10-22 15:06:33.724502 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/management/commands/check.py",
line 65, in handle
2019-10-22 15:06:33.724599 | ubuntu-bionic | fail_level=getattr(checks,
options['fail_level']),
2019-10-22 15:06:33.724810 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/management/base.py",
line 364, in check
2019-10-22 15:06:33.724910 | ubuntu-bionic |
include_deployment_checks=include_deployment_checks,
2019-10-22 15:06:33.725129 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/management/base.py",
line 351, in _run_checks
2019-10-22 15:06:33.725205 | ubuntu-bionic | return
checks.run_checks(**kwargs)
2019-10-22 15:06:33.725421 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/checks/registry.py",
line 73, in run_checks
2019-10-22 15:06:33.725509 | ubuntu-bionic | new_errors =
check(app_configs=app_configs)
2019-10-22 15:06:33.725731 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/checks/urls.py",
line 13, in check_url_config
2019-10-22 15:06:33.725804 | ubuntu-bionic | return check_resolver(resolver)
2019-10-22 15:06:33.726021 | ubuntu-bionic | File
"/home/zuul/src/opendev.org/openstack/horizon/.tox/py36/lib/python3.6/site-packages/django/core/checks/urls.py",
line 23, in check_resolver
2019-10-22
[Yahoo-eng-team] [Bug 1816034] Re: Ironic flavor migration and default resource classes
** Changed in: nova Assignee: Matt Riedemann (mriedem) => (unassigned) ** Changed in: nova Status: In Progress => Triaged ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/train Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1816034 Title: Ironic flavor migration and default resource classes Status in OpenStack Compute (nova): Triaged Status in OpenStack Compute (nova) rocky series: Triaged Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: New Bug description: The Ironic flavor migration to use resource classes happened in Pike/Queens. The flavors and the instances needed to be upgraded with the correct resource class. This was done by an online data migration. Looking into Rocky code: ironic.driver._pike_flavor_migration There is also an offline data migration using nova-manage. These migrations added the node resource class into instance_extra.flavor however I don't see that they also included the default resource classes (VCPU, MEMORY_MB, DISK_GB) set to 0. Looking into Rocky code there is also a TODO in _pike_flavor_migration: "This code can be removed in Queens, and will need to be updated to also alter extra_specs to zero-out the old-style standard resource classes of VCPU, MEMORY_MB, and DISK_GB." Currently all my Ironic instances have the correct node resource class defined, but "old" instances (created before the flavor migration) don't have VCPU, MEMORY_MB, DISK_GB set to 0, in instance_extra.flavor. In Rocky the resource tracker raises the following message: "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Inventory for 'VCPU' on resource provider invalid. ", "title": "Conflict" because it tries to update the allocation but the inventory doesn't have vcpu resources. --- As mitigation we now have: "requires_allocation_refresh = False" in the Ironic Driver. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1816034/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
