Hi Salman, This has turned into a bit of fun -- I'm seeing a lot of wacky things.
First, I'm pretty this issue is local to multi-regions and doesn't have to do with having both Cinder v1 and v2 in the Keystone catalog. I changed my catalog to only have Cinder v2 and I still see multi-region issues. If I had more time, I would figure out how to forcefully make nova-api use Cinder v1 to confirm it's not a v2 issue, but I'm pretty confident that it is not. Second, strange things happen depending on the UUID of the endpoints. Let's say I create two cinder v2 regions: openstack endpoint create --region RegionOne volumev2 --publicurl http://10.1.0.112:8776/v2/%\(tenant_id\)s --internalurl http://10.1.0.112:8776/v2/%\(tenant_id\)s --adminurl http://10.1.0.112:8776/v2/%\(tenant_id\)s +--------------+-----------------------------------------+ | Field | Value | +--------------+-----------------------------------------+ | adminurl | http://10.1.0.112:8776/v2/%(tenant_id)s | | id | a46a5f86b0134944b66c25a7802f7b32 | | internalurl | http://10.1.0.112:8776/v2/%(tenant_id)s | | publicurl | http://10.1.0.112:8776/v2/%(tenant_id)s | | region | RegionOne | | service_id | 9cb6eeba4ae5484080ef1a5272b03367 | | service_name | cinderv2 | | service_type | volumev2 | +--------------+-----------------------------------------+ openstack endpoint create --region RegionTwo volumev2 --publicurl http://10.1.0.113:8776/v2/%\(tenant_id\)s --internalurl http://10.1.0.113:8776/v2/%\(tenant_id\)s --adminurl http://10.1.0.113:8776/v2/%\(tenant_id\)s +--------------+-----------------------------------------+ | Field | Value | +--------------+-----------------------------------------+ | adminurl | http://10.1.0.113:8776/v2/%(tenant_id)s | | id | e9cd7a3fd8734b12a77154d73990261d | | internalurl | http://10.1.0.113:8776/v2/%(tenant_id)s | | publicurl | http://10.1.0.113:8776/v2/%(tenant_id)s | | region | RegionTwo | | service_id | 9cb6eeba4ae5484080ef1a5272b03367 | | service_name | cinderv2 | | service_type | volumev2 | +--------------+-----------------------------------------+ Note the IDs: RegionOne has a46... and RegionTwo has e9c... When I do "openstack endpoint show cinderv2", RegionTwo will be displayed. Why? Because "e" comes after "a". If I regenerate the endpoints until RegionOne has a UUID that is alphabetically later than RegionTwo, then RegionOne will be displayed. It does not matter what my OS_REGION_NAME environment variable is set to, either. Now, let's say my nova.conf file for the nova-api server in RegionOne does not have the [cinder] os_region_name = RegionOne setting I mentioned earlier (also, note that this needs to go in a "[cinder]" section). If RegionOne's UUID comes earlier in the alphabet (so RegionOne would NOT be displayed in "openstack endpoint show cinderv2"), then "nova volume-attach" works. nova-api will grab the correct endpoint for RegionOne. If RegionOne's UUID comes later in the alphabet, then "nova volume-attach" will try to contact RegionTwo. My guess is that the openstack client is reporting the last returned catalog entry while the nova-api service is using the first returned catalog entry. BUT, if I set [cinder] os_region_name = RegionOne then "nova volume-attach" works, no matter what order the entries are in the catalog. Given all of the above, here is what is working for me: 1. I have two services in my Keystone catalog: cinder: volume cinderv2: volumev2 It looks like you have 4: cinder: volume cinderv2: volumev2 cinder-hpc2n: volume cinderv2-hpc2n: volumev2 I don't know if that makes a difference, but maybe simplify the services to just the standard 2. 2. nova.conf on the nova-api server has the "os_region_name" setting set to the correct local region. I only had to restart the nova-api service to see changes take effect, but maybe restart nova-* -- who knows at this point! 3. Check the spelling of your regions. I noticed that you sometimes have "regionOne" and other times have "RegionOne" but maybe that is just a typo from sanitizing your data. Let me know if this helps :) Joe On Thu, Dec 10, 2015 at 2:31 AM, Salman Toor <[email protected]> wrote: > Hi Joe, > > Thanks for your reply, yes Me and Marco are working together. > > I have tried to used > > os_region_name = RegionOne > > both in cinder.conf and nova.conf but no luck so far. But the error > message is a bit different. It seems like now the system is recognising the > region. > > > 2015-12-10 09:32:54.661 26792 TRACE nova.api.openstack EndpointNotFound: > internalURL endpoint for volume service named cinder in regionOne region > not found > > Following are my endpoints: > > —— > > *[root@controller: ~]* # openstack endpoint show cinder > > +--------------+------------------------------------------------+ > > | Field | Value | > > +--------------+------------------------------------------------+ > > | adminurl | http://cinder:8776/v1/%(tenant_id)s | > > | enabled | True | > > | id | e2693bcaf3da4be2810f04acd7995d7f | > > | internalurl | http://cinder:8776/v1/%(tenant_id)s | > > | publicurl | http://cinder:8776/v1/%(tenant_id)s | > > | region | regionOne | > > | service_id | 7bd5d667ec4b4d65b6c1b0de8b303fe3 | > > | service_name | cinder | > > | service_type | volume | > > +--------------+------------------------------------------------+ > > *[root@controller: ~]* # openstack endpoint show cinderv2 > > +--------------+------------------------------------------------+ > > | Field | Value | > > +--------------+------------------------------------------------+ > > | adminurl | http://cinder:8776/v2/%(tenant_id)s | > > | enabled | True | > > | id | 3000cb23c6ab4ee5b68876ee08257338 | > > | internalurl | http://cinder:8776/v2/%(tenant_id)s | > > | publicurl | http://cinder:8776/v2/%(tenant_id)s | > > | region | regionOne | > > | service_id | 61d8baeb4ee74c7798a60758b2f4171f | > > | service_name | cinderv2 | > > | service_type | volumev2 | > > +--------------+------------------------------------------------+ > > *[root@controller: ~]* # openstack endpoint show cinder-hpc2n > > +--------------+--------------------------------------------------+ > > | Field | Value | > > +--------------+--------------------------------------------------+ > > | adminurl | http://cinder:8776/v1/%(tenant_id)s | > > | enabled | True | > > | id | c088f1f9ca0248a08eb60d4ac8492150 | > > | internalurl | http://cinder:8776/v1/%(tenant_id)s | > > | publicurl | http://cinder:8776/v1/%(tenant_id)s | > > | region | HPC2N | > > | service_id | 473f3eab043f4130b30d9ee64fe43703 | > > | service_name | cinder-hpc2n | > > | service_type | volume | > > +--------------+--------------------------------------------------+ > > *[root@controller: ~]* # openstack endpoint show cinderv2-hpc2n > > +--------------+--------------------------------------------------+ > > | Field | Value | > > +--------------+--------------------------------------------------+ > > | adminurl | http://cinder:8776/v2/%(tenant_id)s | > > | enabled | True | > > | id | 985e5b8b068e4c3d99c12788236ee0fe | > > | internalurl | http://cinder:8776/v2/%(tenant_id)s | > > | publicurl | http://cinder:8776/v2/%(tenant_id)s | > > | region | HPC2N | > > | service_id | 089186fad4d7444b95e74ea9d7769423 | > > | service_name | cinderv2-hpc2n | > > | service_type | volumev2 | > > > +--------------+--------------------------------------------------+ > —— > > Again if I remove hpc2n everything started to work fine. Can you tell me > what service I suppose to restart (if required)? > > Regards.. > Salman > > > > PhD, Scientific Computing > Researcher, IT Department, > Uppsala University. > Senior Cloud Architect, > SNIC. > Cloud Application Expert, > UPPMAX. > Visiting Researcher, > Helsinki Institute of Physics (HIP). > [email protected] > http://www.it.uu.se/katalog/salto690 > > On 09 Dec 2015, at 17:48, Joe Topjian <[email protected]> wrote: > > Hi Salman, > > Someone mentioned this same issue yesterday in relation to Terraform > (maybe a colleague of yours?), so given the two occurrences, I thought I'd > look into this. > > I have a Liberty environment readily available, so I created a second set > of volume and volumev2 endpoints for a fictional region. Everything worked > as expected, so I started reviewing the config files and saw that > /etc/cinder/cinder.conf had an option > > [DEFAULT] > os_region_name = RegionOne > > I commented that out, but things still worked. > > Then in /etc/nova/nova.conf, I saw: > > [cinder] > os_region_name = RegionOne > > commenting this out caused volume attachments to hang indefinitely because > nova was trying to contact cinder at RegionTwo (I'm assuming this is the > first catalog entry that was returned). > > Given this is a Liberty environment, it's not accurately reproducing your > problem, but could you check and see if you have that option set in > nova.conf? > > I have a Kilo environment in the process of building. Once it has > finished, I'll see if I can reproduce your error there. > > Thanks, > Joe > > On Wed, Dec 9, 2015 at 4:35 AM, Salman Toor <[email protected]> wrote: > >> Hi, >> >> I am using Kilo release on CentOS. We have recently enabled multiple >> regions and it seems that Cinder have some problems with multiple >> endpoints. >> >> Thinks are working fine with nova but cinder is behaving strange. Here >> are my endpoints >> >> >> ———— >> *[root@controller: ~]* # openstack service list >> +----------------------------------+----------------+--------------+ >> | ID | Name | Type | >> +----------------------------------+----------------+--------------+ >> | 0a33e6f259794ff2a99e626be37c0c2b | cinderv2-hpc2n | volumev2 | >> | 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute | >> | 2c7828120c294d3f82e3a17835babb85 | neutron | network | >> | 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute | >> | 478eff4e96464ae8a958ba29f750b14c | glance | image | >> | 4a5a771d915e43c28e66538b8bc6e625 | cinder | volume | >> | 72d1be82b2e5478dbf0f3fb9e7ba969d | cinderv2 | volumev2 | >> | 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image | >> | a985795b49e2440db82970b81248c86e | cinder-hpc2n | volume | >> | dccd39b92ab547ddaf9047b38620145a | swift | object-store | >> | ebb1660d1d9746759a48de921521bfad | keystone | identity | >> +----------------------------------+----------------+--------------+ >> >> *[root@controller: ~]* # openstack endpoint >> show a985795b49e2440db82970b81248c86e >> +--------------+--------------------------------------------------+ >> | Field | Value | >> +--------------+--------------------------------------------------+ >> | adminurl | http://<URL>:8776/v1/%(tenant_id)s | >> | enabled | True | >> | id | d4003e91ddf24cfb9fa497da81b01a18 | >> | internalurl | http://<URL>:8776/v1/%(tenant_id)s | >> | publicurl | http://<URL>:8776/v1/%(tenant_id)s | >> | region | HPC2N | >> | service_id | a985795b49e2440db82970b81248c86e | >> | service_name | cinder-hpc2n | >> | service_type | volume | >> +--------------+--------------------------------------------------+ >> >> *[root@controller: ~]* # openstack endpoint >> show 4a5a771d915e43c28e66538b8bc6e625 >> +--------------+------------------------------------------------+ >> | Field | Value | >> +--------------+------------------------------------------------+ >> | adminurl | http://<URL>:8776/v1/%(tenant_id)s | >> | enabled | True | >> | id | 5f19c0b535674dbd9e318c7b6d61b3bc | >> | internalurl | http://<URL>:8776/v1/%(tenant_id)s | >> | publicurl | http://<URL>:8776/v1/%(tenant_id)s | >> | region | regionOne | >> | service_id | 4a5a771d915e43c28e66538b8bc6e625 | >> | service_name | cinder | >> | service_type | volume | >> +--------------+------------------------------------------------+ >> >> And same for v2 endpoints >> >> ———— >> >> ——— nova-api.log ——— >> >> achmentController object at 0x598a3d0>>, body: {"volumeAttachment": >> {"device": "", "volumeId": "93d96eab-e3fd-4131-9549-ed51e7299da2"}} >> _process_stack >> /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:780 >> 2015-12-08 12:52:05.847 3376 INFO >> nova.api.openstack.compute.contrib.volumes >> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1 >> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] Attach volume >> 93d96eab-e3fd-4131-9549-ed51e7299da2 to instance >> 3a4c8722-52a7-48f2-beb7-db8938698a0d at >> 2015-12-08 12:52:05.847 3376 DEBUG nova.compute.api >> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1 >> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] [instance: >> 3a4c8722-52a7-48f2-beb7-db8938698a0d] Fetching instance by UUID get >> /usr/lib/python2.7/site-packages/nova/compute/api.py:1911 >> 2015-12-08 12:52:05.988 3376 ERROR nova.api.openstack >> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1 >> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] Caught error: internalURL endpoint >> for volume service named cinder not found >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack Traceback (most >> recent call last): >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line >> 125, in __call__ >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> req.get_response(self.application) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack application, >> catch_exc_info=False) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in >> call_application >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack app_iter = >> application(self.environ, start_response) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> resp(environ, start_response) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", >> line 634, in __call__ >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> self._call_app(env, start_response) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", >> line 554, in _call_app >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> self._app(env, _fake_start_response) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> resp(environ, start_response) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> resp(environ, start_response) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in >> __call__ >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack response = >> self.app(environ, start_response) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> resp(environ, start_response) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack resp = >> self.call_func(req, *args, **self.kwargs) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> self.func(req, *args, **kwargs) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in >> __call__ >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack content_type, >> body, accept) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in >> _process_stack >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack action_result = >> self.dispatch(meth, request, action_args) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in >> dispatch >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> method(req=request, **action_args) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/volumes.py", >> line 305, in create >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume_id, >> device) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 230, in wrapped >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> func(self, context, target, *args, **kwargs) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 219, in inner >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> function(self, context, instance, *args, **kwargs) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 200, in inner >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return f(self, >> context, instance, *args, **kw) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3035, in >> attach_volume >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack disk_bus, >> device_type) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3016, in >> _attach_volume >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack >> volume_bdm.destroy() >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in >> __exit__ >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack >> six.reraise(self.type_, self.value, self.tb) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3009, in >> _attach_volume >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume = >> self.volume_api.get(context, volume_id) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in >> wrapper >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack res = >> method(self, ctx, volume_id, *args, **kwargs) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 286, in get >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack item = >> cinderclient(context).volumes.get(volume_id) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 115, in >> cinderclient >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack url = >> _SESSION.get_endpoint(auth, **service_parameters) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 635, in >> get_endpoint >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> auth.get_endpoint(self, **kwargs) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", >> line 219, in get_endpoint >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack >> service_name=service_name) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in >> inner >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >> func(*args, **kwargs) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >> "/usr/lib/python2.7/site-packages/keystoneclient/service_catalog.py", line >> 254, in url_for >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack raise >> exceptions.EndpointNotFound(msg) >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack EndpointNotFound: >> internalURL endpoint for volume service named cinder not found >> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack >> >> ——— >> >> If I remove cinder-hpc2n and cinderv2-hpc2n from the service list, >> everything started to work fine. >> >> This is how I have created endpoints: >> >> >> openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s >> --internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl >> http://<URL>:8776/v1/%\(tenant_id\)s >> --region regionOne 4a5a771d915e43c28e66538b8bc6e625 >> >> openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s >> --internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl >> http://<URL>:8776/v1/%\(tenant_id\)s >> --region HPC2N a985795b49e2440db82970b81248c86e >> >> using the service-id. >> >> >> Anyone experiencing this behavior? or Any suggestion how to fix it? Again >> regions are working fine with nova and glance. >> >> >> Regards.. >> Salman. >> >> >> PhD, Scientific Computing >> Researcher, IT Department, >> Uppsala University. >> Senior Cloud Architect, >> SNIC. >> Cloud Application Expert, >> UPPMAX. >> Visiting Researcher, >> Helsinki Institute of Physics (HIP). >> [email protected] >> http://www.it.uu.se/katalog/salto690 >> >> >> _______________________________________________ >> OpenStack-operators mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> >> > >
_______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
