[Yahoo-eng-team] [Bug 1758063] Re: neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest failing on networking-midonet gate
** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1758063 Title: neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest failing on networking-midonet gate Status in neutron: Fix Released Bug description: eg. http://logs.openstack.org/13/551013/2/check/networking-midonet- tempest-aio-ml2-full/c661b05/logs/testr_results.html.gz To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1758063/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1758919] Re: Static routes are not per-interface, which breaks some deployments
IMHO, this should also be fixed in cloud-init. If the input netplan contains "global" routes, the renderer (or whatever can pre-process the Netplan before renderering) should intelligently determine which interfaces have an on-link gateway that matches the global route, and automatically render the route at interface scope instead of "global". Arguably, if the route's gateway address doesn't match an on-link prefix, it should not be installed anyway (the kernel will reject it anyway, unless the `onlink` flag is supplied, which instructs the kernel to assume the address is on-link even if it doesn't appear to be). But the only useful scenario I can see for supporting the `onlink` flag is if we're installing a route on an interface that will get is IP address via DHCP. ** Also affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1758919 Title: Static routes are not per-interface, which breaks some deployments Status in cloud-init: New Status in MAAS: In Progress Status in MAAS 2.3 series: Triaged Bug description: When juju tries to deploy a lxd container on a maas managed machine, it looses all static routes, due to ifdown/ifup being issued and e/n/i has no saved data on the original state. Machine with no lxd container deployed: root@4-compute-4:~# ip r default via 100.68.4.254 dev bond2 onlink 100.68.4.0/24 dev bond2 proto kernel scope link src 100.68.4.1 100.68.5.0/24 via 100.68.4.254 dev bond2 100.68.6.0/24 via 100.68.4.254 dev bond2 100.84.4.0/24 dev bond1 proto kernel scope link src 100.84.4.2 100.84.5.0/24 via 100.84.4.254 dev bond1 100.84.6.0/24 via 100.84.4.254 dev bond1 100.99.4.0/24 dev bond0 proto kernel scope link src 100.99.4.101 100.99.5.0/24 via 100.99.4.254 dev bond0 100.99.6.0/24 via 100.99.4.254 dev bond0 100.107.0.0/24 via 100.99.4.254 dev bond0 After juju deploys a container, routes are disappearing: root@4-management-1:~# ip r default via 100.68.100.254 dev bond2 onlink 10.177.144.0/24 dev lxdbr0 proto kernel scope link src 10.177.144.1 100.68.100.0/24 dev bond2 proto kernel scope link src 100.68.100.26 100.84.4.0/24 dev br-bond1 proto kernel scope link src 100.84.4.1 100.99.4.0/24 dev br-bond0 proto kernel scope link src 100.99.4.3 After host reboot, the routes are NOT getting back in place, they are still gone: root@4-management-1:~# ip r s default via 100.68.100.254 dev bond2 onlink 100.68.100.0/24 dev bond2 proto kernel scope link src 100.68.100.26 100.84.4.0/24 dev br-bond1 proto kernel scope link src 100.84.4.1 100.84.5.0/24 via 100.84.4.254 dev br-bond1 100.84.6.0/24 via 100.84.4.254 dev br-bond1 100.99.4.0/24 dev br-bond0 proto kernel scope link src 100.99.4.3 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1758919/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1716448] Re: Enable GVRP for vlan interfaces with linuxbridge agent option
[Expired for neutron because there has been no activity for 60 days.] ** Changed in: neutron Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1716448 Title: Enable GVRP for vlan interfaces with linuxbridge agent option Status in neutron: Expired Bug description: GARP VLAN registration protocol (GVRP) exchanges network VLAN information to allow switches to dynamically forward frames for one or more VLANs. By enabling gvrp on vlan interfaces created by linuxbridge agent operators will be able to dynamically create and destroy vlan based tenant networks. No additional switch configuration or software defined networking is required and brings the features of linuxbridge more in line with openvswitch based clouds. This should be enabled via an option in the linuxbridge agent config; however, there are no serious consequences for having it wrongly enabled. The changes required in the agent are checking the option, if true append 'gvrp on' to the 'ip link add' command that creates the vlan interface. For example 'ip link add link bond0 name bond0.365 type vlan id 365 gvrp on' creates a sub interface for vlan 365 on interface bond0 with gvrp enabled. Adding this capability greatly simplifies switch configuration and deployment of linuxbridge based clouds with minimal impact. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1716448/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1741810] Re: Filter AggregateImagePropertiesIsolation doesn't Work
[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.] ** Changed in: nova Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1741810 Title: Filter AggregateImagePropertiesIsolation doesn't Work Status in OpenStack Compute (nova): Expired Bug description: Description === I tried to use filter AggregateImagePropertiesIsolation to isolate Windows instance for reducing number of Windows Licenses. I think nova scheduler in pike release, filter AggregateImagePropertiesIsolation always returned all hosts. If this is a bug, filter AggregateImagePropertiesIsolation needs to be fixed. Steps to reproduce == # add filter to nova.conf and restart nova scheduler [filter_scheduler] enabled_filters = AggregateImagePropertiesIsolation,RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter # image create with os property openstack image create --min-disk 3 --min-ram 512 --disk-format qcow2 --public --file windows.img img_windows openstack image create --min-disk 1 --min-ram 64 --disk-format qcow2 --public --file cirros-0.3.5-x86_64-disk.img img_linux openstack image set --property os=windows img_windows openstack image set --property os=linux img_linux # host aggregate create with os property openstack aggregate create os_win openstack aggregate add host os_win compute01 openstack aggregate add host os_win compute02 openstack aggregate set --property os=windows os_win openstack aggregate create os_linux openstack aggregate add host os_linux compute03 openstack aggregate add host os_linux compute04 openstack aggregate add host os_linux compute05 openstack aggregate set --property os=linux os_linux # create flavor openstack flavor create --ram 1024 --disk 1 --vcpus 1 --public small openstack flavor create --ram 4096 --disk 20 --vcpus 2 --public medium # create windows instances openstack server create --image img_windows --network test-net --flavor medium --max 10 test-win Expected result === Windows instances can be found in compute01, compute02 only Actual result = Windows instance was found in every hosts. Environment === 1. Nova's version (nova-scheduler)[nova@control01 /]$ rpm -qa | grep nova python-nova-17.0.0-0.20171206190932.cbdc893.el7.centos.noarch openstack-nova-scheduler-17.0.0-0.20171206190932.cbdc893.el7.centos.noarch openstack-nova-common-17.0.0-0.20171206190932.cbdc893.el7.centos.noarch python2-novaclient-9.1.0-0.20170804194758.0a53d19.el7.centos.noarch 2. hypervisor (nova-libvirt)[root@compute01 /]# rpm -qa | grep kvm qemu-kvm-common-ev-2.9.0-16.el7_4.11.1.x86_64 libvirt-daemon-kvm-3.2.0-14.el7_4.5.x86_64 qemu-kvm-ev-2.9.0-16.el7_4.11.1.x86_64 2. Storage ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable) 3. Networking Neutron with OpenVSwitch Logs & Configs == $ tail -f nova-scheduler.log | grep AggregateImagePropertiesIsolation 2018-01-08 11:52:53.964 6 DEBUG nova.filters [req-3828686f-1d46-407a-bebb-14f7a573c52e 9b1f4f0bcea2428c93b8b4276ba67cb7 188be4011b2b49529cbdd6eade152233 - default default] Filter AggregateImagePropertiesIsolation returned 5 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 # add filter to nova.conf and restart nova scheduler [filter_scheduler] enabled_filters = AggregateImagePropertiesIsolation,RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1741810/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1634825] Re: cloud-init rewrites sshd host keys when reboot occurs in GCE
[Expired for cloud-init because there has been no activity for 60 days.] ** Changed in: cloud-init Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1634825 Title: cloud-init rewrites sshd host keys when reboot occurs in GCE Status in cloud-init: Expired Bug description: Hi, Instance SSHd host keys are rewritten by cloud-init when GCE reboot my VM. I have seen that the instance-id has changed after GCE has moved my VM from a dead hypervisor to a new one. I checked /var/lib/cloud/instances/ and can confirm instance-id changed between reboots. We may be able to define in cloud-init configuration what is the idempotent identifier of a server such as the hostname which does not change between reboots. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1634825/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1745320] Re: cloud-init warning message about missing datasource
[Expired for cloud-init because there has been no activity for 60 days.] ** Changed in: cloud-init Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1745320 Title: cloud-init warning message about missing datasource Status in cloud-init: Expired Bug description: When logging in to my Ubuntu instance on Google Cloud I receive the following error message: ~~~ ** # A new feature in cloud-init identified possible datasources for# # this system as:# # ['None'] # # However, the datasource used was: GCE # ## # In the future, cloud-init will only attempt to use datasources that# # are identified or specifically configured. # # For more information see # # https://bugs.launchpad.net/bugs/1669675 # ## # If you are seeing this message, please file a bug against # # cloud-init at # #https://bugs.launchpad.net/cloud-init/+filebug?field.tags=dsid # # Make sure to include the cloud provider your instance is # # running on.# ## # After you have filed a bug, you can disable this warning by launching # # your instance with the cloud-config below, or putting that content # # into /etc/cloud/cloud.cfg.d/99-warnings.cfg# ## # #cloud-config # # warnings: # # dsid_missing_source: off # ** ~~~ Disabling the message as described works, but I don't think that's the way to solve this problem. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1745320/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1759120] [NEW] Objects are not returned if domain name is used instead of domain id
Public bug reported:
# OS_USERNAME=user OS_USER_DOMAIN_NAME=admin_domain OS_PROJECT_NAME=admin
# OS_PROJECT_DOMAIN_NAME=admin_domain
openstack user list --domain testdomain -> users returned for testdomain
# OS_USERNAME=user OS_USER_DOMAIN_NAME=testdomain OS_DOMAIN_NAME=testdomain +
policy file modification
openstack user list --domain 49a912df2669410faecc6e0ab5d8dc80 -> users returned
for testdomain
openstack user list --domain testdomain -> no users returned for testdomain
The same is valid for projects and roles. Role assignments have slightly
different policy rules in a sample file.
Environment: OpenStack Pike (UCA) + a slightly modified
https://github.com/openstack/keystone/blob/stable/pike/etc/policy.v3cloudsample.json
file:
https://paste.ubuntu.com/p/Zk7S7d7qm2/
"admin_and_matching_domain_id": "rule:admin_required and
(domain_id:%(domain_id)s or domain_name:%(domain_id)s)",
domain_name:%(domain_id)s - this was added to allow usage of --domain
, not just ID as documented, e.g. here
https://docs.openstack.org/python-openstackclient/pike/cli/command-
objects/user.html#cmdoption-user-create-domain ("--domain
Default domain (name or ID)")
https://paste.ubuntu.com/p/D35vMMbdTm/ - the first part of this is a
demonstration that a policy file is not enough to use --domain without policy file modification in a non-admin project, the
second part is a demonstration of the problem after policy file
modification.
The domain_name is taken from auth_context and matched against domain_id
API call argument as described here
https://docs.openstack.org/keystone/pike/admin/identity-service-api-
protection.html
Debug mode traces for 3 different scenarios:
https://paste.ubuntu.com/p/8ntVt69tYy/
I can see that the whole Admin scoping and policy enforcement implementation is
being reworked [0][1][2][3] and UUID tokens were deprecated in Pike so
"domain_name" usage from auth context is not a reliable thing to do [4]. If my
understanding is correct, please duplicate or "won't fix" this and let this be
a reference for others to look at. Usage of --domain argument with a domain
name instead of a domain_id is a bit inconsistent in how it's documented in OSC
docs because it seems to only work for the admin user with admin project scoped
tokens (provided that sample policy files are used).
[0] pad.lv/1750673
[1] https://review.openstack.org/#/c/526203/
[2]
https://specs.openstack.org/openstack/keystone-specs/specs/keystone/ongoing/role-check-from-middleware.html
[3] pad.lv/968696
[4] https://review.openstack.org/#/c/525325/
** Affects: keystone
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1759120
Title:
Objects are not returned if domain name is used instead of domain id
Status in OpenStack Identity (keystone):
New
Bug description:
# OS_USERNAME=user OS_USER_DOMAIN_NAME=admin_domain OS_PROJECT_NAME=admin
# OS_PROJECT_DOMAIN_NAME=admin_domain
openstack user list --domain testdomain -> users returned for testdomain
# OS_USERNAME=user OS_USER_DOMAIN_NAME=testdomain OS_DOMAIN_NAME=testdomain +
policy file modification
openstack user list --domain 49a912df2669410faecc6e0ab5d8dc80 -> users
returned for testdomain
openstack user list --domain testdomain -> no users returned for testdomain
The same is valid for projects and roles. Role assignments have
slightly different policy rules in a sample file.
Environment: OpenStack Pike (UCA) + a slightly modified
https://github.com/openstack/keystone/blob/stable/pike/etc/policy.v3cloudsample.json
file:
https://paste.ubuntu.com/p/Zk7S7d7qm2/
"admin_and_matching_domain_id": "rule:admin_required and
(domain_id:%(domain_id)s or domain_name:%(domain_id)s)",
domain_name:%(domain_id)s - this was added to allow usage of --domain
, not just ID as documented, e.g. here
https://docs.openstack.org/python-openstackclient/pike/cli/command-
objects/user.html#cmdoption-user-create-domain ("--domain
Default domain (name or ID)")
https://paste.ubuntu.com/p/D35vMMbdTm/ - the first part of this is a
demonstration that a policy file is not enough to use --domain
without policy file modification in a non-admin project,
the second part is a demonstration of the problem after policy file
modification.
The domain_name is taken from auth_context and matched against
domain_id API call argument as described here
https://docs.openstack.org/keystone/pike/admin/identity-service-api-
protection.html
Debug mode traces for 3 different scenarios:
https://paste.ubuntu.com/p/8ntVt69tYy/
I can see that the whole Admin scoping and policy enforcement implementation
is being reworked [0][1][2][3] and UUID tokens were deprecated in Pike so
"domain_name" usage from auth context is not a reliable thing to do [4]. If my
understanding is c
[Yahoo-eng-team] [Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
This bug was fixed in the package livecd-rootfs - 2.515 --- livecd-rootfs (2.515) bionic; urgency=medium * Set the default locale to C.UTF-8 in all server and cloud images. (LP: #1751051, #1759003) -- Michael Hudson-Doyle Tue, 27 Mar 2018 09:59:02 +1300 ** Changed in: livecd-rootfs (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars Status in cloud-init: Fix Committed Status in cloud-init package in Ubuntu: Fix Released Status in livecd-rootfs package in Ubuntu: Fix Released Bug description: I was testing subiquity, and at the user creation prompt typed in "André D'Silva" for the username, and just "andre" for the login. The installer finished fine, but upon first login I couldn't login. Booting into rescue mode showed me that the user had not been created. Checking cloud-init logs, I find the UnicodeEncodeError. 2018-02-22 12:44:01,386 - __init__.py[DEBUG]: Adding user andre 2018-02-22 12:44:01,387 - util.py[WARNING]: Failed to create user andre 2018-02-22 12:44:01,387 - util.py[DEBUG]: Failed to create user andre Traceback (most recent call last): File "/usr/lib/python3/dist-packages/cloudinit/distros/__init__.py", line 463, in add_user util.subp(adduser_cmd, logstring=log_adduser_cmd) File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1871, in subp env=env, shell=shell) File "/usr/lib/python3.6/subprocess.py", line 709, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child restore_signals, start_new_session, preexec_fn) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 4: ordinal not in range(128) user-data contains this: #cloud-config hostname: sbqt users: - gecos: "Andr\xE9 D'Silva" groups: [adm, cdrom, dip, lpadmin, plugdev, sambashare, debian-tor, libvirtd, lxd, sudo] lock-passwd: false name: andre passwd: $6$UaxxahbQam4Ko1g7$WB5tNuCR84DvWwI7ovxDiofIdLP47pG2USPel2iIQV/qzzT3pAb1VtlbelCR2iCNRxCoJgsVafcNtqdfz1/IL1 shell: /bin/bash ssh_import_id: ['lp:ahasenack'] cloud-init is 17.2-34-g644048e3-0ubuntu1 from bionic/main. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1759100] [NEW] Horizon filtering not thread-safe
Public bug reported: The Horizon filtering is not fully thread-safe. If two users are simultaneously filtering a resource (e.g. projects or instances), and threading is enabled (e.g. 2 or more threads in Apache). It's possible that the wrong filter will display in the form input. I was able to reproduce it using devstack by logging in to Horizon with Admin in Chrome, and Demo in Firefox and spam filter by Project name with "Admin" for the admin user, and "Demo" for the Demo user. This would eventually cause one of the windows to display the other users filter. ** Affects: horizon Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1759100 Title: Horizon filtering not thread-safe Status in OpenStack Dashboard (Horizon): New Bug description: The Horizon filtering is not fully thread-safe. If two users are simultaneously filtering a resource (e.g. projects or instances), and threading is enabled (e.g. 2 or more threads in Apache). It's possible that the wrong filter will display in the form input. I was able to reproduce it using devstack by logging in to Horizon with Admin in Chrome, and Demo in Firefox and spam filter by Project name with "Admin" for the admin user, and "Demo" for the Demo user. This would eventually cause one of the windows to display the other users filter. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1759100/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1758359] Re: nova set-password fails if password already set
These APIs have been this way it seems since they were added (a long
time ago):
https://blueprints.launchpad.net/nova/+spec/get-password
It looks like the only way you can POST a new password for the instance
via the metadata API is if you first reset the password using the DELETE
method via the compute REST API (not the metadata service).
Presumably clearing the password is admin-only by default for security
purposes, i.e. so some other tenant user can't reset the password for
your instance and then post a new password for your guest to hack into
it.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1758359
Title:
nova set-password fails if password already set
Status in OpenStack Compute (nova):
Invalid
Bug description:
If the nova password has been set, trying to set it again (with the
purpose of re-setting the password) fails. Both the nova set-password
command (couldn't find the counterpart in the openstack server help)
as posting the password from inside the instance.
This code seems to not have a retry, if the password is set it returns
an error
if meta_data.password:
raise exc.HTTPConflict()
https://github.com/openstack/nova/blob/master/nova/api/metadata/password.py#L65
I'm running libvirt with KVM/qemu on Ocata. This bug is not related:
https://bugs.launchpad.net/nova/+bug/1757061, that is the effect that
happens after a password set fails.
Could this be changed to allow password changing/resetting if a
password has already been set? For example by accepting an HTTP DELETE
request or allowing an empty password to trigger the reset? ('')
The api does have such an endpoint but it's admin-only by default:
https://developer.openstack.org/api-ref/compute/#clear-admin-password
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1758359/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1758112] Re: Unable to launch Centos/CirrOS instance
There are likely failures in the nova-compute and/or nova-conductor
logs, you would need to dig through those logs (or attach them) to help
diagnose the reason for the build failures.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1758112
Title:
Unable to launch Centos/CirrOS instance
Status in OpenStack Compute (nova):
Invalid
Bug description:
Whenever I try to launch an instance (Centos or CirrOS), it fails with
an Error
fault| {"message": "Exceeded maximum number
of retries. Exhausted all hosts available for retrying build failures for
instance 1f2bdbed-4a34-4ca3-93ed-5a8e0245609e.", "code": 500, "details": "
File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 580,
in build_instances
nova show CirrOS-2
+--+--+
| Property | Value
|
+--+--+
| OS-DCF:diskConfig| MANUAL
|
| OS-EXT-AZ:availability_zone | nova
|
| OS-EXT-SRV-ATTR:host | -
|
| OS-EXT-SRV-ATTR:hostname | cirros-2
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | -
|
| OS-EXT-SRV-ATTR:instance_name| instance-0008
|
| OS-EXT-SRV-ATTR:kernel_id|
|
| OS-EXT-SRV-ATTR:launch_index | 0
|
| OS-EXT-SRV-ATTR:ramdisk_id |
|
| OS-EXT-SRV-ATTR:r
[Yahoo-eng-team] [Bug 1758260] Re: Create instance throws TypeError: argument of type 'NoneType' is not iterable in nova-conductor
This is the error from the nova-conductor logs: 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server [req-0c91a675-d8e1-4e94-86e8-953a30a082bb c327309640084a138cc26e41ec014bb8 4ab0b5b8aecf4096b32a93a5e954b8a5 - default default] Exception during message handling: TypeError: argument of type 'NoneType' is not iterable 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 1268, in schedule_and_build_instances 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server limits=host.limits, host_list=host_list) 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 1234, in build_and_run_instance 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server cctxt.cast(ctxt, 'build_and_run_instance', **kwargs) 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 152, in cast 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server self.transport._send(self.target, msg_ctxt, msg, retry=self.retry) 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 131, in _send 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server timeout=timeout, retry=retry) 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server retry=retry) 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 520, in _send 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server with self._get_connection(rpc_common.PURPOSE_SEND) as conn: 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 475, in _get_connection 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server purpose=purpose) 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/common.py", line 399, in __init__ 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server self.connection = connection_pool.get() 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/pool.py", line 107, in get 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server return self.create() 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/pool.py", line 144, in create 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server return self.connection_cls(self.conf, self.url, purpose) 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 525, in __init__ 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server self._parse_url_hostname(host.hostname) or '', 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 676, in _parse_url_hostname 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server return '[%s]' % hostname if ':' in hostname else hostname 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server TypeError: argument of type 'NoneType' is not iterable 2018-03-23 16:01:30.287 2213 ERROR oslo_messaging.rpc.server Looking at your list_cells output, the transport URL for cell1 is incomplete (there is no host specified): rabbit://openstack:@ So that's likely the problem. The nova-conductor service is trying to RPC cast to cell1 but the transport URL is incomplete. ** Changed in: nova Status: New => Invalid --
[Yahoo-eng-team] [Bug 1758295] Re: nova image-list command display (HTTP 500)
openstack image list is fine, so communicating with glance isn't a
problem.
nova image-list is a proxy to glance, so nova.conf would have to be
configured to communicate with glance, but nova image-list is also
deprecated.
What version of python-novaclient are you using?
And you're running mitaka for the nova server code? In that case, you'd
need to have [glance]/api_servers configured in nova.conf, see:
https://docs.openstack.org/nova/pike/configuration/config.html#glance.api_servers
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1758295
Title:
nova image-list command display (HTTP 500)
Status in OpenStack Compute (nova):
Invalid
Bug description:
Hi,
I need your support here. I have Installed
"contrail-install-packages_3.2.0.0-19-ubuntu-14-04mitaka_all" on a Virtual Box
ubuntu 14.04 server edition.
The installation went fine as per the document I followed and I get into the
web GUI of Openstack Horizon and Contrail. I have 12G RAM/ 4 vCPU's and 100Gig
disk space
I am able to upload cirros image and create networking in Openstack. But When
I try to create a instance in Openstack Horizon it throws an error saying
"error unable to create the server".
I then went to the CLI and display nova flavor-list I am able to see the
different flavors. But when I try to run nova 'image-list' I get errors
bmenezes@contrailsys:~$ nova flavor-list
+++---+--+---+--+---+-+---+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor
| Is_Public |
+++---+--+---+--+---+-+---+
| 1 | m1.tiny| 512 | 1| 0 | | 1 | 1.0
| True |
| 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0
| True |
| 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0
| True |
| 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0
| True |
| 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0
| True |
| 6 | web-flavor | 128 | 1| 0 | | 1 | 1.0
| True |
+++---+--+---+--+---+-+---+
I am able to create a new flavor list as seen ID 6.
bmenezes@contrailsys:~$ nova image-list
ERROR (ClientException): Unexpected API Error. Please report this at
http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
(HTTP 500) (Request-ID:
req-d8494875-7e03-4fda-ac4c-16fa41b01eb0)
bmenezes@contrailsys:~$
Below is the --debug image list
bmenezes@contrailsys:~$ openstack --debug image list
START with options: ['--debug', 'image', 'list']
options: Namespace(access_token_endpoint='', auth_type='',
auth_url='http://127.0.0.1:5000/v2.0', cacert='', client_id='',
client_secret='***', cloud='', debug=True, default_domain='default',
deferred_help=False, domain_id='', domain_name='', endpoint='',
identity_provider='', identity_provider_url='', insecure=None, interface='',
log_file=None, os_clustering_api_version='1', os_compute_api_version='',
os_data_processing_api_version='1.1', os_data_processing_url='',
os_dns_api_version='2', os_identity_api_version='', os_image_api_version='',
os_key_manager_api_version='1', os_network_api_version='',
os_object_api_version='', os_orchestration_api_version='1', os_project_id=None,
os_project_name=None, os_queues_api_version='1.1', os_volume_api_version='',
os_workflow_api_version='2', password='***', profile=None,
project_domain_id='', project_domain_name='', project_id='',
project_name='admin', protocol='', region_name='', scope='',
service_provider_endpoint='', timing=False, token='***', trust_id='', url='',
user_domain_id='', user_domain_name='', user_id='', username='admin',
verbose_level=3, verify=None)
defaults: {u'auth_type': 'password', u'compute_api_version': u'2', 'key':
None, u'database_api_version': u'1.0', 'api_timeout': None,
u'baremetal_api_version': u'1', u'image_api_version': u'2', 'cacert': None,
u'image_api_use_tasks': False, u'floating_ip_source': u'neutron',
u'orchestration_api_version': u'1', u'interface': None, u'network_api_version':
u'2', u'image_format': u'qcow2', u'key_manager_api_version': u'v1',
u'metering_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0',
u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron',
u'container_api_version': u'1', u'dns_api_version': u'2',
u'object_store_api_version': u'1', u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', u'compute_api_version': u'2',
u'orchestration_api_version': '1', u'database_api_version': u'1.0',
'data_pro
[Yahoo-eng-team] [Bug 1758486] Re: nova cant attach volume, unathorized
This likely means you have:
[service_user]
send_service_user_token = True
configured in nova.conf, but you don't have valid credentials configured
in that group for the service user, see:
https://docs.openstack.org/nova/latest/configuration/config.html
#service-user
** No longer affects: cinder
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1758486
Title:
nova cant attach volume, unathorized
Status in OpenStack Compute (nova):
Invalid
Bug description:
ater upgrade to queens, nova unable to attach volume from cinder.
```
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi
[req-6cb77dfe-f718-42d5-a83a-10fa80dea989 fa4ca618dd5247a0841adeac574b54d6
7265d9424e8e4719aa192b08b6d0227b - default default] Unexpected exception in API
method: Unauthorized: The request you have made requires authentication. (HTTP
401)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi Traceback (most
recent call last):
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 788, in
wrapped
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi return
f(*args, **kwargs)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108,
in wrapper
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108,
in wrapper
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/volumes.py", line
336, in create
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi
supports_multiattach=supports_multiattach)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 203, in inner
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi return
function(self, context, instance, *args, **kwargs)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 151, in inner
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi return
f(self, context, instance, *args, **kw)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3940, in
attach_volume
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi volume =
self.volume_api.get(context, volume_id)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 291, in wrapper
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi res =
method(self, ctx, *args, **kwargs)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 313, in wrapper
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi res =
method(self, ctx, volume_id, *args, **kwargs)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 379, in get
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi context,
microversion=microversion).volumes.get(volume_id)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 308,
in get
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi return
self._get("/volumes/%s" % volume_id, "volume")
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/cinderclient/base.py", line 321, in _get
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi resp, body =
self.api.client.get(url)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 199, in
get
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi return
self._cs_request(url, 'GET', **kwargs)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 190, in
_cs_request
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi return
self.request(url, method, **kwargs)
2018-03-24 09:24:12.781 23797 ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 176, in
request
2018-03-24 0
[Yahoo-eng-team] [Bug 1718356] Re: Include default config files in python wheel
Reviewed: https://review.openstack.org/553463 Committed: https://git.openstack.org/cgit/openstack/cyborg/commit/?id=ac6b70dc6a9c440250b739c9fed9a1b74d7642be Submitter: Zuul Branch:master commit ac6b70dc6a9c440250b739c9fed9a1b74d7642be Author: Nguyen Van Trung Date: Thu Mar 15 23:20:29 2018 +0700 Add default configuration files to data_files In order to make it simpler to use the default configuration files when deploying services from source, the files are added to pbr's data_files section so that the files are included in the built wheels and therefore deployed with the code. Packaging and deployment tools can then more easily use the default files if they wish to. This pattern is already established with similar files for neutron and the glance metadefs as has been mentioned in the related bug report. Change-Id: I466f235fec7be024f654739a31365724eaf24097 Closes-Bug: #1718356 ** Changed in: cyborg Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1718356 Title: Include default config files in python wheel Status in Barbican: Fix Released Status in Cinder: Fix Released Status in Cyborg: Fix Released Status in Designate: Fix Released Status in Fuxi: New Status in Glance: Fix Released Status in OpenStack Heat: In Progress Status in Ironic: Fix Released Status in Karbor: In Progress Status in OpenStack Identity (keystone): Fix Released Status in kuryr-libnetwork: New Status in Magnum: In Progress Status in neutron: Fix Released Status in OpenStack Compute (nova): In Progress Status in octavia: Invalid Status in openstack-ansible: Confirmed Status in Sahara: Fix Released Status in OpenStack DBaaS (Trove): Fix Released Status in Zun: In Progress Bug description: The projects which deploy OpenStack from source or using python wheels currently have to either carry templates for api-paste, policy and rootwrap files or need to source them from git during deployment. This results in some rather complex mechanisms which could be radically simplified by simply ensuring that all the same files are included in the built wheel. A precedence for this has already been set in neutron [1], glance [2] and designate [3] through the use of the data_files option in the files section of setup.cfg. [1] https://github.com/openstack/neutron/blob/d3c393ff6b5fbd0bdaabc8ba678d755ebfba08f7/setup.cfg#L24-L39 [2] https://github.com/openstack/glance/blob/02cd5cba70a8465a951cb813a573d390887174b7/setup.cfg#L20-L21 [3] https://github.com/openstack/designate/blob/25eb143db04554d65efe2e5d60ad3afa6b51d73a/setup.cfg#L30-L37 This bug will be used for a cross-project implementation of patches to normalise the implementation across the OpenStack projects. Hopefully the result will be a consistent implementation across all the major projects. A mailing list thread corresponding to this standard setting was begun: http://lists.openstack.org/pipermail/openstack-dev/2017-September/122794.html To manage notifications about this bug go to: https://bugs.launchpad.net/barbican/+bug/1718356/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1759035] [NEW] Unexpected API error
Public bug reported: Hi, Get an error message when I run the command: "openstack compute service list - nova-compute service" command referring to the openstack documentation. the error output message : Erreur d'API inattendue. Signalez-la sur http://bugs.launchpad.net/nova/ et joignez-y le journal de l'API Nova si possible. (HTTP 500) (Request-ID: req-846c6a4a-a513-4814-bddb-391350283f36) " I run Openstack environnement on Queens and my version of nova is : nova --version 9.1.1 I join the nova-api.log Thanks a lot for your help ** Affects: nova Importance: Undecided Status: New ** Tags: nova-api ** Attachment added: "nova-api.log" https://bugs.launchpad.net/bugs/1759035/+attachment/5091574/+files/nova-api.log -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1759035 Title: Unexpected API error Status in OpenStack Compute (nova): New Bug description: Hi, Get an error message when I run the command: "openstack compute service list - nova-compute service" command referring to the openstack documentation. the error output message : Erreur d'API inattendue. Signalez-la sur http://bugs.launchpad.net/nova/ et joignez-y le journal de l'API Nova si possible. (HTTP 500) (Request-ID: req-846c6a4a-a513-4814-bddb-391350283f36) " I run Openstack environnement on Queens and my version of nova is : nova --version 9.1.1 I join the nova-api.log Thanks a lot for your help To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1759035/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1759004] [NEW] No Errors or Warnings In Neutron When Nova API Microversions Are Not Supported
Public bug reported: Throughout OpenStack, the eventlet library is used heavily for threading. In most cases, we use spawn_n() from eventlet which does not return whether the method that was wrapped with it raised an exception. Exceptions are instead just printed to stderr. Many deployers are using simple logging configurations that just log everything to a file. This poses a problem because spawn_n() does not use standard Python logging, but rather just prints a stack trace to stderr, like this: https://github.com/eventlet/eventlet/blob/master/eventlet/greenpool.py#L93 Any exceptions that get thrown in enventlet.spawn_n() are not shown in log files that are setup with normal log configurations that write to files. Deployers who have not been heavily involved with a lot of upstream development or do not know the internals of eventlet would think that there are no errors when tailing their log files which makes debugging problems extremely difficult. There are a few options to make error handling better 1. Use spawn() instead of spawn_n(). spawn() returns a greenthread object that would allow us to see the results of the call. We could check the results and log a proper exception if we used spawn(). The problem is, this would incur a performance penalty as spawn() is slower than spawn_n() 2. Expect deployers to write a custom log handler that will redirect stderr to their log file 3. Use proper exception handling inside of the functions that are called with spawn_n() so that exceptions we care about get caught and logged correctly A specific case where this happens and causes a problem is here: https://github.com/openstack/neutron/blob/7a722159f3bed10fbb26c4d0146f252da72ca0cd/neutron/services/segments/plugin.py#L222 In this scenario we reference aggregate.uuid. However if Nova API does not support microversion 2.41, then this will raise an AttributeError. However, the AttributeError will not be seen by most deployers because it will go to stderr rather than the log files setup with the Python logging framework. Then we start to see strange behavior everywhere and we have no idea why because our logs do not show anything. Logging and exception handling needs to be spruced up in this case and others so that eventlet does not devour important exceptions. Ultimately, we need to be much more cautious of exception handling inside of methods that are called through eventlet. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1759004 Title: No Errors or Warnings In Neutron When Nova API Microversions Are Not Supported Status in neutron: New Bug description: Throughout OpenStack, the eventlet library is used heavily for threading. In most cases, we use spawn_n() from eventlet which does not return whether the method that was wrapped with it raised an exception. Exceptions are instead just printed to stderr. Many deployers are using simple logging configurations that just log everything to a file. This poses a problem because spawn_n() does not use standard Python logging, but rather just prints a stack trace to stderr, like this: https://github.com/eventlet/eventlet/blob/master/eventlet/greenpool.py#L93 Any exceptions that get thrown in enventlet.spawn_n() are not shown in log files that are setup with normal log configurations that write to files. Deployers who have not been heavily involved with a lot of upstream development or do not know the internals of eventlet would think that there are no errors when tailing their log files which makes debugging problems extremely difficult. There are a few options to make error handling better 1. Use spawn() instead of spawn_n(). spawn() returns a greenthread object that would allow us to see the results of the call. We could check the results and log a proper exception if we used spawn(). The problem is, this would incur a performance penalty as spawn() is slower than spawn_n() 2. Expect deployers to write a custom log handler that will redirect stderr to their log file 3. Use proper exception handling inside of the functions that are called with spawn_n() so that exceptions we care about get caught and logged correctly A specific case where this happens and causes a problem is here: https://github.com/openstack/neutron/blob/7a722159f3bed10fbb26c4d0146f252da72ca0cd/neutron/services/segments/plugin.py#L222 In this scenario we reference aggregate.uuid. However if Nova API does not support microversion 2.41, then this will raise an AttributeError. However, the AttributeError will not be seen by most deployers because it will go to stderr rather than the log files setup with the Python logging framework. Then we start to see strange behavior everywhere and we have no idea why because our logs do not show
[Yahoo-eng-team] [Bug 1757472] Re: Required to define database/connection when running services for nova_api cell
The problem is this code is not cell-aware: https://github.com/openstack/nova/blob/2ecb99939ec15057904d1b86c4478def29e193db/nova/api/openstack/wsgi_app.py#L48 It should lookup the cell0 mapping and then use that context for finding the service record entry and creating it in the cell0 DB. ** Changed in: nova Status: New => Triaged ** Changed in: nova Importance: Undecided => Medium ** Also affects: nova/pike Importance: Undecided Status: New ** Also affects: nova/queens Importance: Undecided Status: New ** Changed in: nova/pike Status: New => Triaged ** Changed in: nova/queens Importance: Undecided => Medium ** Changed in: nova/queens Status: New => Triaged ** Changed in: nova/pike Importance: Undecided => Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1757472 Title: Required to define database/connection when running services for nova_api cell Status in OpenStack Compute (nova): Triaged Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Compute (nova) queens series: Triaged Bug description: Services in nova_api cell fail to run if database/connection is not defined. These services should only use api_database/connection. In devstack database/connection is defined with the cell0 DB endpoint. This shouldn't be required because the cell0 is set in nova_api DB. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1757472/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1758952] [NEW] DHCP agent fails to configure routed networks due to failure generating subnet options
Public bug reported: In a routed networks environment, after the merge of https://review.openstack.org/#/c/468744, the dhcp agent attempts to generate subnet options for both local and non local subnets. If one of the non-local subnets is classified as isolated from the point of view of the metadata service, the following traceback occurs: Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 40dcd8c6-e4a1-4510-ac62-f37c11e65a5a.: KeyError: u'2cbde018-1c03-40f8-9d67-90a03c7ebd37' Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent Traceback (most recent call last): Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 144, in call_driver Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs) Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 219, in enable Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent self.spawn_process() Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 446, in spawn_process Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent self._spawn_or_reload_process(reload_with_HUP=False) Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 455, in _spawn_or_reload_process Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent self._output_config_files() Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 499, in _output_config_files Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent self._output_opts_file() Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 872, in _output_opts_file Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent options, subnet_index_map = self._generate_opts_per_subnet() Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 933, in _generate_opts_per_subnet Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent subnet_dhcp_ip and subnet.ip_version == 4): Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent KeyError: u'2cbde018-1c03-40f8-9d67-90a03c7ebd37' Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent This traceback is due to the fact that the non-local subnet in question has no interface in the DHCP agent namespace, so it is not found when: subnet_dhcp_ip = subnet_to_interface_ip[subnet.id] is executed ** Affects: neutron Importance: Medium Assignee: Miguel Lavalle (minsel) Status: New ** Changed in: neutron Assignee: (unassigned) => Miguel Lavalle (minsel) ** Changed in: neutron Importance: Undecided => Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1758952 Title: DHCP agent fails to configure routed networks due to failure generating subnet options Status in neutron: New Bug description: In a routed networks environment, after the merge of https://review.openstack.org/#/c/468744, the dhcp agent attempts to generate subnet options for both local and non local subnets. If one of the non-local subnets is classified as isolated from the point of view of the metadata service, the following traceback occurs: Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 40dcd8c6-e4a1-4510-ac62-f37c11e65a5a.: KeyError: u'2cbde018-1c03-40f8-9d67-90a03c7ebd37' Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent Traceback (most recent call last): Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 144, in call_driver Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs) Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 219, in enable Mar 26 15:21:41 compute2 neutron-dhcp-agent[31181]: ERROR neutron.agent.dhcp.agent self.spawn_process() Mar 26 15:21:
[Yahoo-eng-team] [Bug 1758943] [NEW] Image Import greates taskflow for impossible import scenarios
Public bug reported: When the image_import call is made the taskflow is created and responsible handling the rest of the request. It does not do any prechecks if the import conditions are met (like verifying that the image status transition would be possible. This will create lots of tasks that will end up to the database with no apparent reason. We should make at least basic status checks (like if "glance-direct" method was called agains non "uploading" status or "web-download" method used against non "queued" image) before creating the taskflow. ** Affects: glance Importance: High Status: New ** Affects: glance/queens Importance: High Status: New ** Affects: glance/rocky Importance: High Status: New ** Changed in: glance Importance: Undecided => High ** Also affects: glance/queens Importance: Undecided Status: New ** Also affects: glance/rocky Importance: High Status: New ** Changed in: glance/queens Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1758943 Title: Image Import greates taskflow for impossible import scenarios Status in Glance: New Status in Glance queens series: New Status in Glance rocky series: New Bug description: When the image_import call is made the taskflow is created and responsible handling the rest of the request. It does not do any prechecks if the import conditions are met (like verifying that the image status transition would be possible. This will create lots of tasks that will end up to the database with no apparent reason. We should make at least basic status checks (like if "glance-direct" method was called agains non "uploading" status or "web-download" method used against non "queued" image) before creating the taskflow. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1758943/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1679750] Re: Allocations are not cleaned up in placement for instance 'local delete' case
@Henry, if you need to cleanup placement, there is the osc-placement CLI plugin available for some of this: https://docs.openstack.org/osc-placement/latest/index.html i.e. you can remove allocations for a given consumer (deleted instance) if nova failed to cleanup allocations for that instance when it was deleted. That's a workaround until we have a fix for this bug in nova. ** Also affects: nova/queens Importance: Undecided Status: New ** Changed in: nova/queens Status: New => Confirmed ** Changed in: nova/queens Importance: Undecided => Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1679750 Title: Allocations are not cleaned up in placement for instance 'local delete' case Status in OpenStack Compute (nova): Confirmed Status in OpenStack Compute (nova) pike series: Confirmed Status in OpenStack Compute (nova) queens series: Confirmed Bug description: This is semi-related to bug 1661312 for evacuate. This is the case: 1. Create an instance on host A successfully. There are allocation records in the placement API for the instance (consumer for the allocation records) and host A (resource provider). 2. Host A goes down. 3. Delete the instance. This triggers the local delete flow in the compute API where we can't RPC cast to the compute to delete the instance because the nova-compute service is down. So we do the delete in the database from the compute API (local to compute API, hence local delete). The problem is in #3 we don't remove the allocations for the instance from the host A resource provider during the local delete flow. Maybe this doesn't matter while host A is down, since the scheduler can't schedule to it anyway. But if host A comes back up, it will have allocations tied to it for deleted instances. On init_host in the compute service we call _complete_partial_deletion but that's only for instances with a vm_state of 'deleted' but aren't actually deleted in the database. I don't think that's going to cover this case because the local delete code in the compute API calls instance.destroy() which deletes the instance from the database (updates instances.deleted != 0 in the DB so it's "soft" deleted). To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1679750/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1667863] Re: if a subnet has multiple static routes, the network interfaces file is invalid
Hi, Curtin is no longer involved in rendering of Ubuntu network config, so I've marked its task as Invalid. Please attach the network configuration that MAAS sent. Without this we are not able to do much in cloud-init. In order to further investigate this bug we'll need to get some information about the node. Please collect and attach the storage configuration for the node. You can get this information with: * maas 2.0 via cli maas machine get-curtin-config * Web UI: On the node details page in the installation output section at the bottom of the page When you've provided this information, please set back to 'New' for cloud-init. ** Changed in: curtin Status: New => Invalid ** Changed in: cloud-init Status: New => Incomplete -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1667863 Title: if a subnet has multiple static routes, the network interfaces file is invalid Status in cloud-init: Incomplete Status in curtin: Invalid Status in MAAS: Incomplete Bug description: I have multiple subnets, each has an additional custom static route. those subnets are used by different bridges on the same node. example: brAdm (on interface enp9s0) - subnet 172.30.72.128/25 - static route 172.30.72.0/21 gw 172.30.72.129 brPublic (on interface ens9.2002) - subnet 172.30.80.128/25 - static route 172.30.80.0/21 gw 172.30.80.129 the resulting pre-up and post-up lines in /etc/network/interfaces are malformed, which creates then the wrong routing table. It seems the pre-down of one route and the post-up of the next route are not separated by a newline. See below: post-up route add -net 172.30.80.0 netmask 255.255.248.0 gw 172.30.80.129 metric 0 || true pre-down route del -net 172.30.80.0 netmask 255.255.248.0 gw 172.30.80.129 metric 0 || truepost-up route add -net 172.30.72.0 netmask 255.255.248.0 gw 172.30.72.129 metric 0 || true pre-down route del -net 172.30.72.0 netmask 255.255.248.0 gw 172.30.72.129 metric 0 || true Here's the entire resulting network configuration for reference. note that a bunch of other bridge interfaces are created, but not used on this machine, so not configured. cat /etc/network/interfaces auto lo iface lo inet loopback dns-nameservers 172.30.72.130 dns-search r16maas.os maas auto enp9s0 iface enp9s0 inet manual mtu 9000 auto ens9 iface ens9 inet manual mtu 9000 auto brAdm iface brAdm inet static address 172.30.72.132/25 hwaddress ether 08:9e:01:ab:fc:f6 bridge_ports enp9s0 bridge_fd 15 mtu 9000 auto brData iface brData inet manual hwaddress ether 00:02:c9:ce:7c:16 bridge_ports ens9.0 bridge_fd 15 mtu 9000 auto brExt iface brExt inet manual hwaddress ether 00:02:c9:ce:7c:16 bridge_ports ens9.0 bridge_fd 15 mtu 9000 auto brInt iface brInt inet manual hwaddress ether 00:02:c9:ce:7c:16 bridge_ports ens9.0 bridge_fd 15 mtu 9000 auto brPublic iface brPublic inet static address 172.30.80.132/25 gateway 172.30.80.129 hwaddress ether 00:02:c9:ce:7c:16 bridge_ports ens9.0 bridge_fd 15 mtu 9000 auto brStoClu iface brStoClu inet manual hwaddress ether 00:02:c9:ce:7c:16 bridge_ports ens9.0 bridge_fd 15 mtu 9000 auto brStoData iface brStoData inet manual hwaddress ether 00:02:c9:ce:7c:16 bridge_ports ens9.0 bridge_fd 15 mtu 9000 auto brAdm.52 iface brAdm.52 inet manual vlan_id 52 mtu 1500 vlan-raw-device brAdm auto ens9.0 iface ens9.0 inet manual mtu 9000 vlan-raw-device ens9 post-up route add -net 172.30.80.0 netmask 255.255.248.0 gw 172.30.80.129 metric 0 || true pre-down route del -net 172.30.80.0 netmask 255.255.248.0 gw 172.30.80.129 metric 0 || truepost-up route add -net 172.30.72.0 netmask 255.255.248.0 gw 172.30.72.129 metric 0 || true pre-down route del -net 172.30.72.0 netmask 255.255.248.0 gw 172.30.72.129 metric 0 || true source /etc/network/interfaces.d/*.cfg route Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 172.30.72.128 * 255.255.255.128 U 0 00 brAdm 172.30.80.128 * 255.255.255.128 U 0 00 brPublic ifconfig brAdm Link encap:Ethernet HWaddr 08:9e:01:ab:fc:f6 inet addr:172.30.72.132 Bcast:172.30.72.255 Mask:255.255.255.128 inet6 addr: fe80::a9e:1ff:feab:fcf6/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:15029 errors:0 dropped:0 overruns:0 frame:0 TX packets:1447 error
[Yahoo-eng-team] [Bug 1758879] [NEW] cloud-init aws doesn't changing 50-cloud-init.cfg on boot
Public bug reported: Hi there, I notice that using the latest version of cloud-init on ubuntu xenial running on AWS the cloud-init isn't changing the 50-cloud-init.cfg file on boot, that behavior can turn the instance unreachable in some cases. A particular case is when you change the instance type from t2.* to r4.* and try to enable the ENA support, the network changes from eth0 to ens3 and the 50-cloud-init.cfg isn't changed to reflect it and turns the instance unreachable. To identify: Launch an instance on us-east-1 using an old ami (ami-e13739f6) and manually change the file /etc/network/interfaces.d/50-cloud-init.cfg. After a stop/start, the file is regenerated Launch an instance on us-east-1 using the latest ami (ami-66506c1c) and anually change the file /etc/network/interfaces.d/50-cloud-init.cfg. After a stop/start, the file remains the same. I cannot found if it's a bug or an expected behavior, however, it seems a bug as the file has a message saying that "Changes to it will not persist across an instance." ** Affects: cloud-init Importance: Undecided Status: New ** Attachment added: "cloud-init.tar" https://bugs.launchpad.net/bugs/1758879/+attachment/5091075/+files/cloud-init.tar -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1758879 Title: cloud-init aws doesn't changing 50-cloud-init.cfg on boot Status in cloud-init: New Bug description: Hi there, I notice that using the latest version of cloud-init on ubuntu xenial running on AWS the cloud-init isn't changing the 50-cloud-init.cfg file on boot, that behavior can turn the instance unreachable in some cases. A particular case is when you change the instance type from t2.* to r4.* and try to enable the ENA support, the network changes from eth0 to ens3 and the 50-cloud-init.cfg isn't changed to reflect it and turns the instance unreachable. To identify: Launch an instance on us-east-1 using an old ami (ami-e13739f6) and manually change the file /etc/network/interfaces.d/50-cloud-init.cfg. After a stop/start, the file is regenerated Launch an instance on us-east-1 using the latest ami (ami-66506c1c) and anually change the file /etc/network/interfaces.d/50-cloud-init.cfg. After a stop/start, the file remains the same. I cannot found if it's a bug or an expected behavior, however, it seems a bug as the file has a message saying that "Changes to it will not persist across an instance." To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1758879/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
