[Yahoo-eng-team] [Bug 1621818] Re: nova-compute unexpected input/output errors on starting instances (NFS + image-cache)
The patch was proposed against stable/mitaka but needs to be proposed against master first, then backported to stable/newton and finally stable/mitaka if that's your target release. ** Also affects: nova/newton Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1621818 Title: nova-compute unexpected input/output errors on starting instances (NFS + image-cache) Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) mitaka series: In Progress Status in OpenStack Compute (nova) newton series: New Bug description: Our setup consists of multiple controllers and multiple hypervisors. Our shared storage for the instances is on a nfs 4.1 export. Using Ubuntu 16.04 LTS and Openstack Mitaka When we launch an instance, nova updates the mtime for the _base image to let the image cache manager know the image is actively used. I think this was added here: https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=fb6ca3e7c8a38328d384cd41c061ded6623dac90 Because of this, in our setup, we are seeing unexpected input/output errors: Stderr: u"/bin/touch: setting times of '/var/lib/nova/instances/_base/79e34519bacb47ad6f64c4baca4d33fd5c57d34d': Input/output error A full trace can be found here: http://paste.openstack.org/show/570161/ This error particularly shows itself when launching multiple instances at once. Also, because of this error, the instances are rescheduled. The assigned neutron ports, however, are not deleted. This results in multiple ip's assigned to the instances, with only one of them UP. This also results in attached floating ip's not working .. This is similar to https://bugs.launchpad.net/nova/+bug/1609526, nova should tell neutron, either to delete the unused port, or update it instead of creating a new one. Some more info on our environment: -- Using libvirt + kvm, neutron with openvswitch L3 HA # dpkg -l | grep nova ii nova-common2:13.0.0-0ubuntu2 all OpenStack Compute - common files ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM) ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support ii python-nova2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7 # dpkg -l |grep libvirt ii libvirt-bin1.3.1-1ubuntu10.1 amd64programs for the libvirt library ii libvirt0:amd64 1.3.1-1ubuntu10.1 amd64library for interfacing with different virtualization systems ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support ii python-libvirt 1.3.1-1ubuntu1 amd64libvirt Python bindings To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1621818/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1633734] [NEW] ValueError: Field `instance_uuid' cannot be None
Public bug reported: I "accidentally" upgraded from Mitaka to Newton a few days ago and I'm still cleaning up "the mess" that introduced (to used to Debian GNU/Linux packages takes care of all that for me). Anyway, I'm now getting ValueError: Field `instance_uuid' cannot be None in the nova-api log. I've been looking at http://docs.openstack.org/releasenotes/nova/newton.html#upgrade-notes but I'm not sure what to do. I've run nova-manage db online_data_migrations => ERROR nova.db.sqlalchemy.api [req-c08dbccb-d841-4e38-a895-26768f24222b - - - - -] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version nova-manage db sync => ERROR: could not access cell mapping database - has api db been created? nova-manage api_db sync => Seems to run ok nova-manage cell_v2 discover_hosts => error: 'module' object has no attribute 'session' nova-manage cell_v2 map_cell0 => Seemed like it ran ok nova-manage cell_v2 simple_cell_setup --transport-url rabbit://blabla/ => Seemed like it ran ok nova-manage db null_instance_uuid_scan => There were no records found where instance_uuid was NULL. Other than that, I'm not sure what the problem is. ** Affects: nova Importance: Undecided Status: New ** Tags: upgrades -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1633734 Title: ValueError: Field `instance_uuid' cannot be None Status in OpenStack Compute (nova): New Bug description: I "accidentally" upgraded from Mitaka to Newton a few days ago and I'm still cleaning up "the mess" that introduced (to used to Debian GNU/Linux packages takes care of all that for me). Anyway, I'm now getting ValueError: Field `instance_uuid' cannot be None in the nova-api log. I've been looking at http://docs.openstack.org/releasenotes/nova/newton.html#upgrade-notes but I'm not sure what to do. I've run nova-manage db online_data_migrations => ERROR nova.db.sqlalchemy.api [req-c08dbccb-d841-4e38-a895-26768f24222b - - - - -] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version nova-manage db sync => ERROR: could not access cell mapping database - has api db been created? nova-manage api_db sync => Seems to run ok nova-manage cell_v2 discover_hosts => error: 'module' object has no attribute 'session' nova-manage cell_v2 map_cell0 => Seemed like it ran ok nova-manage cell_v2 simple_cell_setup --transport-url rabbit://blabla/ => Seemed like it ran ok nova-manage db null_instance_uuid_scan => There were no records found where instance_uuid was NULL. Other than that, I'm not sure what the problem is. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1633734/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1633735] [NEW] Can't deploy overcloud with ocata/trunk on rhel
Public bug reported: 2016-10-15 03:32:33.655 1933 CRITICAL nova [req-a1b7237a-b12b-4f98-be57-00cdc63ebab1 - - - - -] ConfigFileValueError: Value for option osapi_compute_workers is not valid: Should be greater than or equal to 1 2016-10-15 03:32:33.655 1933 ERROR nova Traceback (most recent call last): 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/bin/nova-api", line 10, in 2016-10-15 03:32:33.655 1933 ERROR nova sys.exit(main()) 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/lib/python2.7/site-packages/nova/cmd/api.py", line 60, in main 2016-10-15 03:32:33.655 1933 ERROR nova server = service.WSGIService(api, use_ssl=should_use_ssl) 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 296, in __init__ 2016-10-15 03:32:33.655 1933 ERROR nova self.workers = (getattr(CONF, '%s_workers' % wname, None) or 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2267, in __getattr__ 2016-10-15 03:32:33.655 1933 ERROR nova return self._get(name) 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2672, in _get 2016-10-15 03:32:33.655 1933 ERROR nova value = self._do_get(name, group, namespace) 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2715, in _do_get 2016-10-15 03:32:33.655 1933 ERROR nova % (opt.name, str(ve))) 2016-10-15 03:32:33.655 1933 ERROR nova ConfigFileValueError: Value for option osapi_compute_workers is not valid: Should be greater than or equal to 1 ** Affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1633735 Title: Can't deploy overcloud with ocata/trunk on rhel Status in cloud-init: New Bug description: 2016-10-15 03:32:33.655 1933 CRITICAL nova [req-a1b7237a-b12b-4f98-be57-00cdc63ebab1 - - - - -] ConfigFileValueError: Value for option osapi_compute_workers is not valid: Should be greater than or equal to 1 2016-10-15 03:32:33.655 1933 ERROR nova Traceback (most recent call last): 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/bin/nova-api", line 10, in 2016-10-15 03:32:33.655 1933 ERROR nova sys.exit(main()) 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/lib/python2.7/site-packages/nova/cmd/api.py", line 60, in main 2016-10-15 03:32:33.655 1933 ERROR nova server = service.WSGIService(api, use_ssl=should_use_ssl) 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 296, in __init__ 2016-10-15 03:32:33.655 1933 ERROR nova self.workers = (getattr(CONF, '%s_workers' % wname, None) or 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2267, in __getattr__ 2016-10-15 03:32:33.655 1933 ERROR nova return self._get(name) 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2672, in _get 2016-10-15 03:32:33.655 1933 ERROR nova value = self._do_get(name, group, namespace) 2016-10-15 03:32:33.655 1933 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2715, in _do_get 2016-10-15 03:32:33.655 1933 ERROR nova % (opt.name, str(ve))) 2016-10-15 03:32:33.655 1933 ERROR nova ConfigFileValueError: Value for option osapi_compute_workers is not valid: Should be greater than or equal to 1 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1633735/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1605804] Re: Instance creation sometimes fails after host aggregate deletion
** Also affects: nova/mitaka Importance: Undecided Status: New ** Changed in: nova/mitaka Assignee: (unassigned) => Roman Podoliaka (rpodolyaka) ** Changed in: nova/mitaka Status: New => In Progress ** Changed in: nova/mitaka Importance: Undecided => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1605804 Title: Instance creation sometimes fails after host aggregate deletion Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) mitaka series: In Progress Bug description: Instance creation starts failing if nova scheduler gets in an inconsistent state wrt host aggregates. If remove_host_from_aggregate operation is invoked for multiple hosts in quick succession, followed by aggregate deletion, the nova scheduler host_manager maps (host_aggregates_map and aggs_by_id) get out of sync, as there are some stale references left behind in the host_aggregates_map for an aggregate that is deleted from the aggs_by_id map. This is because it cleans up state based on aggregate.hosts which is empty when aggregate is deleted, but the prior aggregate updates to remove individual hosts could have incorrect list of hosts added to the host_aggregates_map. Instance creation fails with below error once scheduler gets in this state: 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher [req-7f29701b-0272-444c-8650-a1035777e642 d2c755daa21e451e86c1d2b5be705aa2 0546d7f9c747456aa0ffb306cfe5627d - - -] Exception during message handling: 1 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher incoming.message)) 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs) 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/manager.py", line 84, in select_destinations 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher filter_properties) 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 72, in select_destinations 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher filter_properties) 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 164, in _schedule 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher hosts = self._get_all_host_states(elevated) 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 222, in _get_all_host_states 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher return self.host_manager.get_all_host_states(context) 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 585, in get_all_host_states 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher host_state.host]] 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher KeyError: 1 2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher 2016-07-21 18:20:16.784 15692 ERROR oslo_messaging._drivers.common [req-7f29701b-0272-444c-8650-a1035777e642 d2c755daa21e451e86c1d2b5be705aa2 0546d7f9c747456aa0ffb306cfe5627d - - -] Returning exception 1 to caller To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1605804/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help
[Yahoo-eng-team] [Bug 1590608] Re: Services should use http_proxy_to_wsgi middleware
** Also affects: freezer Importance: Undecided Status: New ** Changed in: freezer Assignee: (unassigned) => Jeremy Liu (liujiong) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1590608 Title: Services should use http_proxy_to_wsgi middleware Status in Aodh: Fix Released Status in Barbican: In Progress Status in Ceilometer: Fix Released Status in Cinder: Fix Released Status in cloudkitty: In Progress Status in congress: New Status in Freezer: In Progress Status in Glance: Fix Released Status in Gnocchi: Fix Committed Status in heat: Fix Released Status in OpenStack Identity (keystone): Fix Released Status in Magnum: New Status in neutron: In Progress Status in Panko: Fix Released Status in OpenStack Search (Searchlight): In Progress Status in senlin: In Progress Status in OpenStack DBaaS (Trove): In Progress Bug description: It's a common problem when putting a service behind a load balancer to need to forward the Protocol and hosts of the original request so that the receiving service can construct URLs to the loadbalancer and not the private worker node. Most services have implemented some form of secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO handling however exactly how this is done is dependent on the service. oslo.middleware provides the http_proxy_to_wsgi middleware that handles these headers and the newer RFC7239 forwarding header and completely hides the problem from the service. This middleware should be adopted by all services in preference to their own HTTP_X_FORWARDED_PROTO handling. To manage notifications about this bug go to: https://bugs.launchpad.net/aodh/+bug/1590608/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1594812] Re: domain + ldap configuration breaks ability to add admin user to admin project
[Expired for OpenStack Identity (keystone) because there has been no activity for 60 days.] ** Changed in: keystone Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1594812 Title: domain + ldap configuration breaks ability to add admin user to admin project Status in OpenStack Identity (keystone): Expired Bug description: Ubuntu 16.04 LTS with Mitaka installation from ubuntu repo packages. All seems to work until I tested keystone using domain configurations + ldap With the following configuration enabled: domain_specific_drivers_enabled = true domain_configurations_from_database = false I am only able to create a role, project and user. When I try using assignment to assign the user to the project with role admin it fails. root@supafly /home/chris $ openstack role add --domain default --user admin admin Could not find resource admin But I was able to successfully create the user and its visible in the LDAP database using the openstack python cli. When I try login with the user admin that I created, i get an error user not assigned to any domains or projects. So I disabled domain_Sepcific_drivers_enabled by setting it to false: domain_specific_drivers_enabled = false I tried to create the user again, which was also succesfully. Then when I tried to assign role it worked fine. However does not work with domain_specific_drivers_enabled. From my understanding is if I remove the domain_specific_configuration file /etc/keystone/keystone_default.conf Then login with domain default then it should not be using LDAP. Since the driver is only set to LDAP within the domain specific configuration. It should then be using SQL. But the results are exactly the same. So its something related to enable the domain_specific_configuration. Please advice what output is necessary. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1594812/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
