[Yahoo-eng-team] [Bug 1692090] Re: _dn_to_id ignores user_id_attribute
[Expired for OpenStack Identity (keystone) because there has been no activity for 60 days.] ** Changed in: keystone Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1692090 Title: _dn_to_id ignores user_id_attribute Status in OpenStack Identity (keystone): Expired Bug description: _dn_to_id is not affected when user_id_attribute is changed in keystone.conf. https://github.com/openstack/keystone/blob/c3ca06ff47cced16ea9de3d6ef1a6c583bb3cf38/keystone/identity/backends/ldap/common.py#L1280 Considering the following LDAP directory: ... # userid, Users, openstack.org dn: cn=userid,ou=Users,dc=openstack,dc=org objectClass: inetOrgPerson userPassword:: e1NTSEF9Rit1bTlOS2FKdWM2bWFhWUtmRGQ5dmlBdEd6NEFydHY= sn: 25cbd5b54da849128b89c3f7ab6e5bff cn: userid # test-group, UserGroups, openstack.org dn: cn=test-group,ou=UserGroups,dc=openstack,dc=org objectClass: groupOfNames cn: test-group ou: f44a7fbb9e174ba5823474c759d43643 member: cn=userid,ou=Users,dc=openstack,dc=org ... keystone.conf: ... user_id_attribute = sn user_name_attribute = cn ... This results in users unable to found in groups. e.g. `openstack user list --domain default --group test-group` Expected: User ID and Name are returned +--+-+ | ID | Name| +--+-+ | 25cbd5b54da849128b89c3f7ab6e5bff | userid | +--+-+ Actual: Nothing is returned To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1692090/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1693704] Re: Unable to list federated projects with unscoped token
[Expired for OpenStack Identity (keystone) because there has been no
activity for 60 days.]
** Changed in: keystone
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1693704
Title:
Unable to list federated projects with unscoped token
Status in OpenStack Identity (keystone):
Expired
Bug description:
When I got the federated user project list, the error is as bellow:
2017-05-26 15:12:54.685 12742 ERROR keystone.common.wsgi File
"/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 228, in
__call__
2017-05-26 15:12:54.685 12742 ERROR keystone.common.wsgi result =
method(req, **params)
2017-05-26 15:12:54.685 12742 ERROR keystone.common.wsgi File
"/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 164, in
inner
2017-05-26 15:12:54.685 12742 ERROR keystone.common.wsgi return f(self,
request, *args, **kwargs)
2017-05-26 15:12:54.685 12742 ERROR keystone.common.wsgi File
"/usr/lib/python2.7/site-packages/keystone/federation/controllers.py", line
480, in list_projects_for_user
2017-05-26 15:12:54.685 12742 ERROR keystone.common.wsgi
request.auth_context['group_ids'])
2017-05-26 15:12:54.685 12742 ERROR keystone.common.wsgi KeyError: 'group_ids'
2017-05-26 15:12:54.685 12742 ERROR keystone.common.wsgi
and I have got the token scoped in domain.
My mapping rule is as bellow:
[
{
"local": [
{
"user": {
"name": "{0}",
"domain": {
"name": "{1}"
},
"type": "local"
}
}
],
"remote": [
{
"type": "openstack_user"
},
{
"type": "openstack_user_domain"
}
]
}
]
The error is that token is an unscoped token which is got from the API
“/v3/OS-FEDERATION/identity_providers/keystone-
idp/protocols/saml2/auth” and then the federated user want to get the
projects from /v3/OS-FEDERATION/projects. But error occurs.
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1693704/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1775947] [NEW] tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest failing
Public bug reported: Since few days I see that tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment in neutron-tempest-dvr job. Example of failure: http://logs.openstack.org/90/572690/2/check/neutron- tempest-dvr/45ec391/logs/testr_results.html.gz It happened at least 3 times on 8.06: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22exit%20status%3A%20137%2C%20stderr%3A%20Killed%5C%22%20AND%20build_name%3A%5C %22neutron-tempest-dvr%5C%22 ** Affects: neutron Importance: Undecided Status: New ** Tags: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1775947 Title: tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest failing Status in neutron: New Bug description: Since few days I see that tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment in neutron-tempest-dvr job. Example of failure: http://logs.openstack.org/90/572690/2/check /neutron-tempest-dvr/45ec391/logs/testr_results.html.gz It happened at least 3 times on 8.06: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22exit%20status%3A%20137%2C%20stderr%3A%20Killed%5C%22%20AND%20build_name%3A%5C %22neutron-tempest-dvr%5C%22 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1775947/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1775934] [NEW] Cannot run "openstack server list" with instance stuck in scheduling state
Public bug reported:
Seeing this in Ocata at 125dd1f30fdaf50182256c56808a5199856383c7.
Running `openstack server list --project
9c28d07207a54c78848fd7b4f85779d5` results in a 500 error:
RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please
report this at http://bugs.launchpad.net/nova/ and attach the Nova API
log if possible.\n", "code":
500}}
Traceback in nova-api:
http://paste.openstack.org/show/6YrSmjMSo0lIxyFjbPIz/
some data on the instance:
http://paste.openstack.org/show/6PSa35HvdxZCQnVQ2sQU/
Looks like lazy-loading the flavor is failing because it's looking in
the wrong database.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1775934
Title:
Cannot run "openstack server list" with instance stuck in scheduling
state
Status in OpenStack Compute (nova):
New
Bug description:
Seeing this in Ocata at 125dd1f30fdaf50182256c56808a5199856383c7.
Running `openstack server list --project
9c28d07207a54c78848fd7b4f85779d5` results in a 500 error:
RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please
report this at http://bugs.launchpad.net/nova/ and attach the Nova API
log if possible.\n", "code":
500}}
Traceback in nova-api:
http://paste.openstack.org/show/6YrSmjMSo0lIxyFjbPIz/
some data on the instance:
http://paste.openstack.org/show/6PSa35HvdxZCQnVQ2sQU/
Looks like lazy-loading the flavor is failing because it's looking in
the wrong database.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1775934/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1775782] Re: glance-image-import.conf not parsed when running under wsgi
Taken into consideration that we don't currently support nor encourage anyone running under wsgi mode either mod_wsgi nor uwsgi , I'd say the priority for this bug is very low, specially as you mentioned in your comment the fix needs to happen in Devstack rather than in Glance. ** Changed in: glance Importance: Undecided => Low ** Changed in: glance Status: New => Opinion -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1775782 Title: glance-image-import.conf not parsed when running under wsgi Status in Glance: Opinion Bug description: When development environment is set to run under wsgi mode (by setting WSGI_MODE=mod_wsgi in local.conf) glance-image-import.conf file is not parsed. It is working properly if glance is running under uwsgi. Steps to reproduce: NOTE: Here I am trying to use plugin "inject_image_metadata" to inject metadata properties to the image. 1. Add glance-image-import.conf as mentioned at /etc/glance [image_import_opts] image_import_plugins = ["inject_image_metadata"] [inject_metadata_properties] inject = "property1":"value" ignore_user_roles = demo 2. Restart glance api service using "devstack@g-api.service" 3. Source using admin credentials (as we are ignoring demo role in glance-image-import.conf) $ source devstack/openrc admin admin 3. Create image using import api $ glance image-create-via-import --container-format bare --disk-format qcow2 --name ceph_image_default --file temp.qcow2 Expected Output: Properties mentioned in glance-image-import.conf should be injected to the image Actual Output: Properties are not injected as plugin is not loaded. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1775782/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1774402] Re: Glance scrubber SELinux denials
This is a bug in RDO packaging rather than bug in Glance. Please file
the bug in RDO [0] and you have much more luck to get it fixed.
[0] https://bugzilla.redhat.com/enter_bug.cgi?product=RDO
** Changed in: glance
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1774402
Title:
Glance scrubber SELinux denials
Status in Glance:
Invalid
Bug description:
Glance scrubber on RHEL7 from RDO with SELinux enabled get denied
connecting to cinder & swift
type=AVC msg=audit(1527765224.059:149655): avc: denied { name_connect } for
pid=1283 comm="glance-scrubber" dest=8776
scontext=system_u:system_r:glance_scrubber_t:s0 tcontext=
system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1527765228.066:149656): avc: denied { name_connect } for
pid=1283 comm="glance-scrubber" dest=8776
scontext=system_u:system_r:glance_scrubber_t:s0 tcontext=
system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1527765228.690:149657): avc: denied { name_connect } for
pid=1283 comm="glance-scrubber" dest=8080
scontext=system_u:system_r:glance_scrubber_t:s0 tcontext=
system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket
Enabling the nis_enabled seboolean allows connections to cinder,
swift looks to need
allow glance_scrubber_t http_cache_port_t:tcp_socket name_connect;
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1774402/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1775879] [NEW] Create a domain, projects, users, and roles in keystone
Public bug reported: When I try to create new project I have get error as shown in the attachment. I had done everything before in the instruction without any problems. What can it be? This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes: - [ ] This doc is inaccurate in this way: __ - [ ] This is a doc addition request. - [ ] I have a fix to the document that I can paste below including example: input and output. If you have a troubleshooting or support issue, use the following resources: - Ask OpenStack: http://ask.openstack.org - The mailing list: http://lists.openstack.org - IRC: 'openstack' channel on Freenode --- Release: 12.0.1.dev18 on 2018-04-21 04:02 SHA: 22af1d9f35c86e9c5bca288c2996be5c19e3cd61 Source: https://git.openstack.org/cgit/openstack/keystone/tree/doc/source/install/keystone-users-ubuntu.rst URL: https://docs.openstack.org/keystone/pike/install/keystone-users-ubuntu.html ** Affects: keystone Importance: Undecided Status: New ** Tags: doc ** Attachment added: "Error report" https://bugs.launchpad.net/bugs/1775879/+attachment/5150316/+files/IMG_20180608_174438.jpg -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1775879 Title: Create a domain, projects, users, and roles in keystone Status in OpenStack Identity (keystone): New Bug description: When I try to create new project I have get error as shown in the attachment. I had done everything before in the instruction without any problems. What can it be? This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes: - [ ] This doc is inaccurate in this way: __ - [ ] This is a doc addition request. - [ ] I have a fix to the document that I can paste below including example: input and output. If you have a troubleshooting or support issue, use the following resources: - Ask OpenStack: http://ask.openstack.org - The mailing list: http://lists.openstack.org - IRC: 'openstack' channel on Freenode --- Release: 12.0.1.dev18 on 2018-04-21 04:02 SHA: 22af1d9f35c86e9c5bca288c2996be5c19e3cd61 Source: https://git.openstack.org/cgit/openstack/keystone/tree/doc/source/install/keystone-users-ubuntu.rst URL: https://docs.openstack.org/keystone/pike/install/keystone-users-ubuntu.html To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1775879/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1775863] [NEW] Cannot save a RequestSpec object if nothing changed
Public bug reported: If a RequestSpec.save() is called on an object that has no changed field the code blows up with UnboundLocalError: File "nova/tests/unit/objects/test_request_spec.py", line 622, in test_save_does_not_persist_requested_resources req_obj.save() File "/mnt/ssd/ebalgib/nova/py27/.tox/py27/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper return fn(self, *args, **kwargs) File "nova/objects/request_spec.py", line 588, in save updates = self._get_update_primitives() File "nova/objects/request_spec.py", line 561, in _get_update_primitives return db_updates UnboundLocalError: local variable 'db_updates' referenced before assignment It is because the db_updates local is only initialized if there is something to update but returned regardless https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L518 https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L532 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1775863 Title: Cannot save a RequestSpec object if nothing changed Status in OpenStack Compute (nova): New Bug description: If a RequestSpec.save() is called on an object that has no changed field the code blows up with UnboundLocalError: File "nova/tests/unit/objects/test_request_spec.py", line 622, in test_save_does_not_persist_requested_resources req_obj.save() File "/mnt/ssd/ebalgib/nova/py27/.tox/py27/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper return fn(self, *args, **kwargs) File "nova/objects/request_spec.py", line 588, in save updates = self._get_update_primitives() File "nova/objects/request_spec.py", line 561, in _get_update_primitives return db_updates UnboundLocalError: local variable 'db_updates' referenced before assignment It is because the db_updates local is only initialized if there is something to update but returned regardless https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L518 https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L532 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1775863/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1775758] Re: Deprecated auth_url entries in Neutron Queen's install guide
Please look at the source url that you linked before filing bugs as this has already been fixed. ** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1775758 Title: Deprecated auth_url entries in Neutron Queen's install guide Status in neutron: Invalid Bug description: This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes: - [x] This doc is inaccurate in this way: The Neutron installation guides use old auth_uri / auth_url values pointing at two different keystone endpoints for authentication of the network service and the compute service. This occurs within the controller node and compute node parts of the installation guide. Following the current guide then fails at the Verify step with error "Failed to retrieve extensions list from Network API" - [x] I have a fix to the document that I can paste below including example: input and output. input 1 (neutron.conf): auth_uri = http://controller:5000 auth_url = http://controller:35357 input 2 (nova.conf): auth_url = http://controller:35357 output: auth_url = http://controller:5000 I changed the values to the above output and the Verify step succeeded. --- Release: 12.0.3.dev24 on 2018-06-07 22:47 SHA: 2206636feca043a9ab958010a00641f92957e8a5 Source: https://git.openstack.org/cgit/openstack/neutron/tree/doc/source/install/controller-install-ubuntu.rst URL: https://docs.openstack.org/neutron/queens/install/controller-install-ubuntu.html To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1775758/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
OK figured this one out - the cephx keys are missing a permission which allows them to see blacklisted clients - as a result they can't deal with a hard crash: mon 'allow command "osd blacklist"' This is a charm issue after all. As a workaround you can manually update the existing client keys for nova-compute using: sudo ceph auth caps client.nova-compute mon 'allow r, allow command "osd blacklist"' osd 'allow rwx' from any mon unit. ** Changed in: nova Status: New => Invalid ** Changed in: ceph (Ubuntu) Status: New => Invalid ** Changed in: nova (Ubuntu) Status: New => Invalid ** Changed in: cloud-archive Status: New => Invalid ** Changed in: qemu (Ubuntu) Status: New => Invalid ** Also affects: charm-ceph-mon Importance: Undecided Status: New ** Also affects: charms.ceph Importance: Undecided Status: New ** Changed in: charms.ceph Status: New => Triaged ** Changed in: charm-ceph-mon Status: New => Triaged ** Changed in: charms.ceph Importance: Undecided => High ** Changed in: charm-ceph-mon Importance: Undecided => High ** Changed in: charm-ceph-mon Milestone: None => 18.08 ** Changed in: cloud-archive Assignee: Sean Feole (sfeole) => (unassigned) ** Changed in: nova (Ubuntu) Assignee: Sean Feole (sfeole) => (unassigned) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage Status in OpenStack ceph-mon charm: Triaged Status in charms.ceph: Triaged Status in Ubuntu Cloud Archive: Invalid Status in OpenStack Compute (nova): Invalid Status in ceph package in Ubuntu: Invalid Status in nova package in Ubuntu: Invalid Status in qemu package in Ubuntu: Invalid Bug description: Reboot host that contains VMs with volumes and all VMs fail to boot. Happens with Queens on Bionic and Xenial [0.00] Initializing cgroup subsys cpuset [0.00] Initializing cgroup subsys cpu [0.00] Initializing cgroup subsys cpuacct [0.00] Linux version 4.4.0-124-generic (buildd@lcy01-amd64-028) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) ) #148-Ubuntu SMP Wed May 2 13:00:18 UTC 2018 (Ubuntu 4.4.0-124.148-generic 4.4.117) [0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-124-generic root=UUID=bca2de6e-f774-4203-ae05-e8deeb05f64a ro console=tty1 console=ttyS0 [0.00] KERNEL supported cpus: [0.00] Intel GenuineIntel [0.00] AMD AuthenticAMD [0.00] Centaur CentaurHauls [0.00] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [0.00] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers' [0.00] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers' [0.00] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers' [0.00] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [0.00] x86/fpu: Using 'eager' FPU context switches. [0.00] e820: BIOS-provided physical RAM map: [0.00] BIOS-e820: [mem 0x-0x0009fbff] usable [0.00] BIOS-e820: [mem 0x0009fc00-0x0009] reserved [0.00] BIOS-e820: [mem 0x000f-0x000f] reserved [0.00] BIOS-e820: [mem 0x0010-0x7ffdbfff] usable [0.00] BIOS-e820: [mem 0x7ffdc000-0x7fff] reserved [0.00] BIOS-e820: [mem 0xfeffc000-0xfeff] reserved [0.00] BIOS-e820: [mem 0xfffc-0x] reserved [0.00] NX (Execute Disable) protection: active [0.00] SMBIOS 2.8 present. [0.00] Hypervisor detected: KVM [0.00] e820: last_pfn = 0x7ffdc max_arch_pfn = 0x4 [0.00] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT [0.00] found SMP MP-table at [mem 0x000f6a20-0x000f6a2f] mapped at [880f6a20] [0.00] Scanning 1 areas for low memory corruption [0.00] Using GB pages for direct mapping [0.00] RAMDISK: [mem 0x361f4000-0x370f1fff] [0.00] ACPI: Early table checksum verification disabled [0.00] ACPI: RSDP 0x000F6780 14 (v00 BOCHS ) [0.00] ACPI: RSDT 0x7FFE1649 2C (v01 BOCHS BXPCRSDT 0001 BXPC 0001) [0.00] ACPI: FACP 0x7FFE14CD 74 (v01 BOCHS BXPCFACP 0001 BXPC 0001) [0.00] ACPI: DSDT 0x7FFE0040 00148D (v01 BOCHS BXPCDSDT 0001 BXPC 0001) [0.00] ACPI: FACS 0x7FFE 40 [0.00] ACPI: APIC 0x7FFE15C1 88 (v01 BOCHS BXPCAPIC 0
[Yahoo-eng-team] [Bug 1775797] [NEW] The mac table size of neutron bridges (br-tun, br-int, br-*) is too small by default and eventually makes openvswitch explode
Public bug reported: Description of problem: the CPU utilization of ovs-vswitchd is high without DPDK enabled PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 1512 root 10 -10 4352840 793864 12008 R 1101 0.3 15810:26 ovs-vswitchd at the same time we were observing failures to send packets (ICMP) over VXLAN tunnel, we think this might be related to high CPU usage. --- Reproducer and analysis on ovs side done by Jiri Benc: Reproducer: Create an ovs bridge: -- ovs-vsctl add-br ovs0 ip l s ovs0 up -- Save this to a file named "reproducer.py": -- #!/usr/bin/python from scapy.all import * data = [(str(RandMAC()), str(RandIP())) for i in range(int(sys.argv[1]))] s = conf.L2socket(iface="ovs0") while True: for mac, ip in data: p = Ether(src=mac, dst=mac)/IP(src=ip, dst=ip) s.send(p) -- Run the reproducer: ./reproducer.py 5000 The problem is how flow revalidation works in ovs. There are several 'revalidator' threads launched. They should normally sleep (modulo waking every 0.5 second just to do nothing) and they wake if anything of interest happens (udpif_revalidator => poll_block). On every wake up, each revalidator thread checks whether flow revalidation is needed and if it is, it does the revalidation. The revalidation is very costly with high number of flows. I also suspect there's a lot of contention between the revalidator threads. The flow revalidation is triggered by many things. What is of interest for us is that any eviction of a MAC learning table entry triggers revalidation. The reproducer script repeatedly sends the same 5000 packets, all of them with a different MAC address. This causes constant overflows of the MAC learning table and constant revalidation. The revalidator threads are being immediately woken up and are busy looping the revalidation. Which is exactly the pattern from the customers' data: there are 16000+ flows and the packet capture shows that the packets are repeating every second. A quick fix is to increase the MAC learning table size: ovs-vsctl set bridge other-config:mac-table-size=5 This should lower the CPU usage down substantially; allow a few seconds for things to settle down. ** Affects: neutron Importance: Medium Assignee: Slawek Kaplonski (slaweq) Status: Confirmed ** Tags: ovs -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1775797 Title: The mac table size of neutron bridges (br-tun, br-int, br-*) is too small by default and eventually makes openvswitch explode Status in neutron: Confirmed Bug description: Description of problem: the CPU utilization of ovs-vswitchd is high without DPDK enabled PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 1512 root 10 -10 4352840 793864 12008 R 1101 0.3 15810:26 ovs-vswitchd at the same time we were observing failures to send packets (ICMP) over VXLAN tunnel, we think this might be related to high CPU usage. --- Reproducer and analysis on ovs side done by Jiri Benc: Reproducer: Create an ovs bridge: -- ovs-vsctl add-br ovs0 ip l s ovs0 up -- Save this to a file named "reproducer.py": -- #!/usr/bin/python from scapy.all import * data = [(str(RandMAC()), str(RandIP())) for i in range(int(sys.argv[1]))] s = conf.L2socket(iface="ovs0") while True: for mac, ip in data: p = Ether(src=mac, dst=mac)/IP(src=ip, dst=ip) s.send(p) -- Run the reproducer: ./reproducer.py 5000 The problem is how flow revalidation works in ovs. There are several 'revalidator' threads launched. They should normally sleep (modulo waking every 0.5 second just to do nothing) and they wake if anything of interest happens (udpif_revalidator => poll_block). On every wake up, each revalidator thread checks whether flow revalidation is needed and if it is, it does the revalidation. The revalidation is very costly with high number of flows. I also suspect there's a lot of contention between the revalidator threads. The flow revalidation is triggered by many things. What is of interest for us is that any eviction of a MAC learning table entry triggers revalidation. The reproducer script repeatedly sends the same 5000 packets, all of them with a different MAC address. This causes constant overflows of the MAC learning table and constant revalidation. The revalidator threads are being immediately woken up and are busy looping the revalidation. Which is exactly the pattern from the customers' data: there are 16000+ flows and the packet capture shows that the packets are repeating every second. A quick fix is to increase the MAC learning table size: ovs-vsctl set bridge other-config:mac-table-size=5 This should lower the CPU usage down s
[Yahoo-eng-team] [Bug 1773449] Re: VMs do not survive host reboot
Raising bug tasks for ceph and qemu as I think this is where the issue lies; nova generates the same libvirt xml disk stanzas for versions that work and versions that have this issue. ** Also affects: ceph (Ubuntu) Importance: Undecided Status: New ** Also affects: qemu (Ubuntu) Importance: Undecided Status: New ** Changed in: ceph (Ubuntu) Importance: Undecided => High ** Changed in: qemu (Ubuntu) Importance: Undecided => High ** Summary changed: - VMs do not survive host reboot + VM rbd backed block devices inconsistent after unexpected host outage -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage Status in Ubuntu Cloud Archive: New Status in OpenStack Compute (nova): New Status in ceph package in Ubuntu: New Status in nova package in Ubuntu: New Status in qemu package in Ubuntu: New Bug description: Reboot host that contains VMs with volumes and all VMs fail to boot. Happens with Queens on Bionic and Xenial [0.00] Initializing cgroup subsys cpuset [0.00] Initializing cgroup subsys cpu [0.00] Initializing cgroup subsys cpuacct [0.00] Linux version 4.4.0-124-generic (buildd@lcy01-amd64-028) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) ) #148-Ubuntu SMP Wed May 2 13:00:18 UTC 2018 (Ubuntu 4.4.0-124.148-generic 4.4.117) [0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-124-generic root=UUID=bca2de6e-f774-4203-ae05-e8deeb05f64a ro console=tty1 console=ttyS0 [0.00] KERNEL supported cpus: [0.00] Intel GenuineIntel [0.00] AMD AuthenticAMD [0.00] Centaur CentaurHauls [0.00] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [0.00] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers' [0.00] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers' [0.00] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers' [0.00] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [0.00] x86/fpu: Using 'eager' FPU context switches. [0.00] e820: BIOS-provided physical RAM map: [0.00] BIOS-e820: [mem 0x-0x0009fbff] usable [0.00] BIOS-e820: [mem 0x0009fc00-0x0009] reserved [0.00] BIOS-e820: [mem 0x000f-0x000f] reserved [0.00] BIOS-e820: [mem 0x0010-0x7ffdbfff] usable [0.00] BIOS-e820: [mem 0x7ffdc000-0x7fff] reserved [0.00] BIOS-e820: [mem 0xfeffc000-0xfeff] reserved [0.00] BIOS-e820: [mem 0xfffc-0x] reserved [0.00] NX (Execute Disable) protection: active [0.00] SMBIOS 2.8 present. [0.00] Hypervisor detected: KVM [0.00] e820: last_pfn = 0x7ffdc max_arch_pfn = 0x4 [0.00] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT [0.00] found SMP MP-table at [mem 0x000f6a20-0x000f6a2f] mapped at [880f6a20] [0.00] Scanning 1 areas for low memory corruption [0.00] Using GB pages for direct mapping [0.00] RAMDISK: [mem 0x361f4000-0x370f1fff] [0.00] ACPI: Early table checksum verification disabled [0.00] ACPI: RSDP 0x000F6780 14 (v00 BOCHS ) [0.00] ACPI: RSDT 0x7FFE1649 2C (v01 BOCHS BXPCRSDT 0001 BXPC 0001) [0.00] ACPI: FACP 0x7FFE14CD 74 (v01 BOCHS BXPCFACP 0001 BXPC 0001) [0.00] ACPI: DSDT 0x7FFE0040 00148D (v01 BOCHS BXPCDSDT 0001 BXPC 0001) [0.00] ACPI: FACS 0x7FFE 40 [0.00] ACPI: APIC 0x7FFE15C1 88 (v01 BOCHS BXPCAPIC 0001 BXPC 0001) [0.00] No NUMA configuration found [0.00] Faking a node at [mem 0x-0x7ffdbfff] [0.00] NODE_DATA(0) allocated [mem 0x7ffd7000-0x7ffdbfff] [0.00] kvm-clock: Using msrs 4b564d01 and 4b564d00 [0.00] kvm-clock: cpu 0, msr 0:7ffcf001, primary cpu clock [0.00] kvm-clock: using sched offset of 17590935813 cycles [0.00] clocksource: kvm-clock: mask: 0x max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [0.00] Zone ranges: [0.00] DMA [mem 0x1000-0x00ff] [0.00] DMA32[mem 0x0100-0x7ffdbfff] [0.00] Normal empty [0.00] Device empty [0.00] Movable zone start for each node [0.00] Early
[Yahoo-eng-team] [Bug 1775782] [NEW] glance-image-import.conf not parsed when running under wsgi
Public bug reported: When development environment is set to run under wsgi mode (by setting WSGI_MODE=mod_wsgi in local.conf) glance-image-import.conf file is not parsed. It is working properly if glance is running under uwsgi. Steps to reproduce: NOTE: Here I am trying to use plugin "inject_image_metadata" to inject metadata properties to the image. 1. Add glance-image-import.conf as mentioned at /etc/glance [image_import_opts] image_import_plugins = ["inject_image_metadata"] [inject_metadata_properties] inject = "property1":"value" ignore_user_roles = demo 2. Restart glance api service using "devstack@g-api.service" 3. Source using admin credentials (as we are ignoring demo role in glance-image-import.conf) $ source devstack/openrc admin admin 3. Create image using import api $ glance image-create-via-import --container-format bare --disk-format qcow2 --name ceph_image_default --file temp.qcow2 Expected Output: Properties mentioned in glance-image-import.conf should be injected to the image Actual Output: Properties are not injected as plugin is not loaded. ** Affects: glance Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1775782 Title: glance-image-import.conf not parsed when running under wsgi Status in Glance: New Bug description: When development environment is set to run under wsgi mode (by setting WSGI_MODE=mod_wsgi in local.conf) glance-image-import.conf file is not parsed. It is working properly if glance is running under uwsgi. Steps to reproduce: NOTE: Here I am trying to use plugin "inject_image_metadata" to inject metadata properties to the image. 1. Add glance-image-import.conf as mentioned at /etc/glance [image_import_opts] image_import_plugins = ["inject_image_metadata"] [inject_metadata_properties] inject = "property1":"value" ignore_user_roles = demo 2. Restart glance api service using "devstack@g-api.service" 3. Source using admin credentials (as we are ignoring demo role in glance-image-import.conf) $ source devstack/openrc admin admin 3. Create image using import api $ glance image-create-via-import --container-format bare --disk-format qcow2 --name ceph_image_default --file temp.qcow2 Expected Output: Properties mentioned in glance-image-import.conf should be injected to the image Actual Output: Properties are not injected as plugin is not loaded. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1775782/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
