Hi Salman, That's awesome news. Glad it's working. :)
Joe On Fri, Dec 11, 2015 at 3:12 PM, Salman Toor <salman.t...@it.uu.se> wrote: > Hi, > > It is working by setting exact names of the services. > > > *[root@smog: ~]* # openstack service list > > +----------------------------------+--------------+--------------+ > > | ID | Name | Type | > > +----------------------------------+--------------+--------------+ > > | 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute | > > | 2c7828120c294d3f82e3a17835babb85 | neutron | network | > > | 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute | > > | 478eff4e96464ae8a958ba29f750b14c | glance | image | > > | 61d8baeb4ee74c7798a60758b2f4171f | cinderv2 | volumev2 | > > | 75f89962c7864507b07055fbfc98053e | cinder | volume | > > | 7bd5d667ec4b4d65b6c1b0de8b303fe3 | cinder | volume | > > | 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image | > > | 9d2a7ef6b36c45b096e552bf73cb89ae | cinderv2 | volumev2 | > > | dccd39b92ab547ddaf9047b38620145a | swift | object-store | > > | ebb1660d1d9746759a48de921521bfad | keystone | identity | > > +----------------------------------+--------------+--------------+ > > *[root@smog: ~]* # openstack endpoint list > > > +----------------------------------+-----------+--------------+--------------+ > > | ID | Region | Service Name | Service > Type | > > > +----------------------------------+-----------+--------------+--------------+ > > | 3000cb23c6ab4ee5b68876ee08257338 | regionOne | cinderv2 | volumev2 > | > > | e21e935ed861484e976d1b93e0fda0f0 | regionOne | nova | compute > | > > | cdfb008c47a0472ab3a93e6ee07e9ba4 | regionOne | neutron | network > | > > | e2693bcaf3da4be2810f04acd7995d7f | regionOne | cinder | volume > | > > | 1400c42cbf154f63b0c3e8d64352d1f2 | HPC2N | cinderv2 | volumev2 > | > > | 04acd1666704433d991cf9a75957c815 | HPC2N | glance-hpc2n | image > | > > | cfd1c766c56744309767cd84034f9bfb | regionOne | swift | > object-store | > > | abe46f0d86064056927a2177a705787c | HPC2N | nova-hpc2n | compute > | > > | 52bb09199ec84ef09642151348eab695 | HPC2N | cinder | volume > | > > | ac522bcd576b4d2b9f5adfb5405730be | regionOne | keystone | identity > | > > | bad524f18fd74ec3b7fb6647ea661686 | regionOne | glance | image > | > > +----------------------------------+-----------+--------------+--------------+ > > > > But different names are working perfectly fine with glance and nova. > > Anyways thanks for all your time and efforts. If I manage to reproduce it > on devstack, will report you back. > > Regards.. > Salman > > > > > ------------------------------ > *From:* Joe Topjian [j...@topjian.net] > *Sent:* Friday, December 11, 2015 7:16 PM > *To:* Salman Toor > *Cc:* openstack-operators@lists.openstack.org > *Subject:* Re: [Openstack-operators] Cinder API with multiple regions not > working. > > Hi Salman, > > I'm stumped. > > I was able to confirm that Keystone acts a little differently when you > have multiple regional endpoints connected to the same "service_name" and > "service_type" than if you have multiple regional endpoints, each with > their own "service_name" but same "service_type"'s. > > For example, you have: > > regionOne, cinder, volume > regionOne, cinderv2, volumev2 > HPC2N, cinderhpc2n, volume > HPC2N, cinderv2hpc2n, volumev2 > > When you do "openstack endpoint show volume", you should not get any > output. But if you do "openstack endpoint show cinderhpc2n", then you will > see output. Swap the service_type and service_names and the pattern > continues. > > However, I am still unable to reproduce your issue. > > My initial tests used the following format: > > regionOne, cinder, volume > regionOne, cinderv2, volumev2 > HPC2N, cinder, volume > HPC2N, cinderv2, volumev2 > > With this format, you will run into the UUID ordering issue I described > previously. But again, I'm not able to reproduce the error you're seeing. > > I really hate to give this answer, but at this point, I'd recommend > setting up a devstack environment and comparing the configuration files > under /etc/nova and /etc/cinder. This problem is not occurring for me in > devstack, so there has to be some difference between the two that is > causing this issue. > > Once you have devstack installed, you can create a fictional second region > by doing: > > Your current style: > > openstack service create --name cinderhpc2n volume > openstack service create --name cinderhpc2nv2 volumev2 > openstack endpoint create --region HPC2N cinderhpc2n --publicurl > http://10.1.0.113:8776/v1/%\(tenant_id\)s --internalurl > http://10.1.0.113:8776/v1/%\(tenant_id\)s --adminurl > http://10.1.0.113:8776/v1/%\(tenant_id\)s > openstack endpoint create --region HPC2N cinderhpc2nv2 --publicurl > http://10.1.0.113:8776/v2/%\(tenant_id\)s --internalurl > http://10.1.0.113:8776/v2/%\(tenant_id\)s --adminurl > http://10.1.0.113:8776/v2/%\(tenant_id\)s > > My style which I was originally testing with: > > openstack endpoint create --region HPC2N volume --publicurl > http://10.1.0.113:8776/v1/%\(tenant_id\)s --internalurl > http://10.1.0.113:8776/v1/%\(tenant_id\)s --adminurl > http://10.1.0.113:8776/v1/%\(tenant_id\)s > openstack endpoint create --region HPC2N volumev2 --publicurl > http://10.1.0.113:8776/v2/%\(tenant_id\)s --internalurl > http://10.1.0.113:8776/v2/%\(tenant_id\)s --adminurl > http://10.1.0.113:8776/v2/%\(tenant_id\)s > > If the IP address is not in use, you will see commands hang when they try > to reach the wrong region. I find this to be an easy troubleshooting method > to discover where problems are being hit. > > If you figure out what the issue is, please let me know as I'm very > curious. > > Joe > > On Fri, Dec 11, 2015 at 3:58 AM, Salman Toor <salman.t...@it.uu.se> wrote: > >> Hi, >> >> Still no luck. I have remove the “-“ from the names. >> >> *[root@controller: ~/openstack]* # openstack service list >> +----------------------------------+---------------+--------------+ >> | ID | Name | Type | >> +----------------------------------+---------------+--------------+ >> | 0f6736013cba469b9ddf21988a071952 | cinderhpc2n | volume | >> | 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute | >> | 2c7828120c294d3f82e3a17835babb85 | neutron | network | >> | 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute | >> | 478eff4e96464ae8a958ba29f750b14c | glance | image | >> | 61d8baeb4ee74c7798a60758b2f4171f | cinderv2 | volumev2 | >> | 7bd5d667ec4b4d65b6c1b0de8b303fe3 | cinder | volume | >> | 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image | >> | dccd39b92ab547ddaf9047b38620145a | swift | object-store | >> | dec231e13a114365b6e094421204255f | cinderhpc2nv2 | volumev2 | >> | ebb1660d1d9746759a48de921521bfad | keystone | identity | >> +----------------------------------+---------------+--------------+ >> *[root@controller: ~/openstack]* # openstack endpoint list >> >> +----------------------------------+-----------+---------------+--------------+ >> | ID | Region | Service Name | Service >> Type | >> >> +----------------------------------+-----------+---------------+--------------+ >> | 3000cb23c6ab4ee5b68876ee08257338 | regionOne | cinderv2 | >> volumev2 | >> | e21e935ed861484e976d1b93e0fda0f0 | regionOne | nova | compute >> | >> | 42211e6e977d43308a4e288e68b97263 | HPC2N | cinderhpc2nv2 | >> volumev2 | >> | cdfb008c47a0472ab3a93e6ee07e9ba4 | regionOne | neutron | network >> | >> | e2693bcaf3da4be2810f04acd7995d7f | regionOne | cinder | volume >> | >> | 04acd1666704433d991cf9a75957c815 | HPC2N | glance-hpc2n | image >> | >> | eb1a12119c5c4b2dba78927982121be7 | HPC2N | cinderhpc2n | volume >> | >> | cfd1c766c56744309767cd84034f9bfb | regionOne | swift | >> object-store | >> | abe46f0d86064056927a2177a705787c | HPC2N | nova-hpc2n | compute >> | >> | ac522bcd576b4d2b9f5adfb5405730be | regionOne | keystone | >> identity | >> | bad524f18fd74ec3b7fb6647ea661686 | regionOne | glance | image >> | >> >> +----------------------------------+-----------+---------------+--------------+ >> *[root@controller: ~/openstack]* # systemctl restart >> openstack-nova-api.service >> >> Even with all the services didn’t help. >> >> *[root@controller: ~/openstack]* # nova volume-attach >> 800d8ba0-cc17-4877-894a-89adecfb5eb7 85ab8b8a-c75c-45a1-9f51-44ed75ba3210 >> /dev/vdb >> ERROR (ClientException): The server has either erred or is incapable of >> performing the requested operation. (HTTP 500) (Request-ID: >> req-f0d9372d-863b-43f6-8281-5c9196a866af) >> *[root@controller: ~/openstack]* # openstack service delete cinderhpc2n >> *[root@controller: ~/openstack]* # openstack service delete cinderhpc2nv2 >> *[root@controller: ~/openstack]* # >> *[root@controller: ~/openstack]* # >> *[root@controller: ~/openstack]* # >> *[root@controller: ~/openstack]* # >> *[root@controller: ~/openstack]* # nova volume-attach >> 800d8ba0-cc17-4877-894a-89adecfb5eb7 85ab8b8a-c75c-45a1-9f51-44ed75ba3210 >> /dev/vdb >> +----------+--------------------------------------+ >> | Property | Value | >> +----------+--------------------------------------+ >> | device | /dev/vdb | >> | id | 85ab8b8a-c75c-45a1-9f51-44ed75ba3210 | >> | serverId | 800d8ba0-cc17-4877-894a-89adecfb5eb7 | >> | volumeId | 85ab8b8a-c75c-45a1-9f51-44ed75ba3210 | >> >> +----------+--------------------------------------+ >> >> >> —— nova-api.log —— >> 2015-12-11 09:43:08.534 16529 TRACE nova.api.openstack EndpointNotFound: >> internalURL endpoint for volume service named cinder in regionOne region >> not found >> —— >> >> ——nova.conf >> [cinder] >> os_region_name = regionOne >> —— >> >> —— cinder.conf >> [default] >> os_region_name = regionOne >> —— >> >> Regards.. >> Salman. >> >> >> >> >> >> PhD, Scientific Computing >> Researcher, IT Department, >> Uppsala University. >> Senior Cloud Architect, >> SNIC. >> Cloud Application Expert, >> UPPMAX. >> Visiting Researcher, >> Helsinki Institute of Physics (HIP). >> salman.t...@it.uu.se >> http://www.it.uu.se/katalog/salto690 >> >> On 10 Dec 2015, at 23:35, Joe Topjian <j...@topjian.net> wrote: >> >> Hi Salman, >> >> This has turned into a bit of fun -- I'm seeing a lot of wacky things. >> >> First, I'm pretty this issue is local to multi-regions and doesn't have >> to do with having both Cinder v1 and v2 in the Keystone catalog. I changed >> my catalog to only have Cinder v2 and I still see multi-region issues. If I >> had more time, I would figure out how to forcefully make nova-api use >> Cinder v1 to confirm it's not a v2 issue, but I'm pretty confident that it >> is not. >> >> Second, strange things happen depending on the UUID of the endpoints. >> Let's say I create two cinder v2 regions: >> >> openstack endpoint create --region RegionOne volumev2 --publicurl >> http://10.1.0.112:8776/v2/%\(tenant_id\)s --internalurl >> http://10.1.0.112:8776/v2/%\(tenant_id\)s --adminurl >> http://10.1.0.112:8776/v2/%\(tenant_id\)s >> >> +--------------+-----------------------------------------+ >> | Field | Value | >> +--------------+-----------------------------------------+ >> | adminurl | http://10.1.0.112:8776/v2/%(tenant_id)s | >> | id | a46a5f86b0134944b66c25a7802f7b32 | >> | internalurl | http://10.1.0.112:8776/v2/%(tenant_id)s | >> | publicurl | http://10.1.0.112:8776/v2/%(tenant_id)s | >> | region | RegionOne | >> | service_id | 9cb6eeba4ae5484080ef1a5272b03367 | >> | service_name | cinderv2 | >> | service_type | volumev2 | >> +--------------+-----------------------------------------+ >> >> openstack endpoint create --region RegionTwo volumev2 --publicurl >> http://10.1.0.113:8776/v2/%\(tenant_id\)s --internalurl >> http://10.1.0.113:8776/v2/%\(tenant_id\)s --adminurl >> http://10.1.0.113:8776/v2/%\(tenant_id\)s >> >> +--------------+-----------------------------------------+ >> | Field | Value | >> +--------------+-----------------------------------------+ >> | adminurl | http://10.1.0.113:8776/v2/%(tenant_id)s | >> | id | e9cd7a3fd8734b12a77154d73990261d | >> | internalurl | http://10.1.0.113:8776/v2/%(tenant_id)s | >> | publicurl | http://10.1.0.113:8776/v2/%(tenant_id)s | >> | region | RegionTwo | >> | service_id | 9cb6eeba4ae5484080ef1a5272b03367 | >> | service_name | cinderv2 | >> | service_type | volumev2 | >> +--------------+-----------------------------------------+ >> >> Note the IDs: RegionOne has a46... and RegionTwo has e9c... >> >> When I do "openstack endpoint show cinderv2", RegionTwo will be >> displayed. Why? Because "e" comes after "a". If I regenerate the endpoints >> until RegionOne has a UUID that is alphabetically later than RegionTwo, >> then RegionOne will be displayed. >> >> It does not matter what my OS_REGION_NAME environment variable is set to, >> either. >> >> Now, let's say my nova.conf file for the nova-api server in RegionOne >> does not have the >> >> [cinder] >> os_region_name = RegionOne >> >> setting I mentioned earlier (also, note that this needs to go in a >> "[cinder]" section). >> >> If RegionOne's UUID comes earlier in the alphabet (so RegionOne would NOT >> be displayed in "openstack endpoint show cinderv2"), then "nova >> volume-attach" works. nova-api will grab the correct endpoint for RegionOne. >> >> If RegionOne's UUID comes later in the alphabet, then "nova >> volume-attach" will try to contact RegionTwo. >> >> My guess is that the openstack client is reporting the last returned >> catalog entry while the nova-api service is using the first returned >> catalog entry. >> >> BUT, if I set >> >> [cinder] >> os_region_name = RegionOne >> >> then "nova volume-attach" works, no matter what order the entries are in >> the catalog. >> >> >> Given all of the above, here is what is working for me: >> >> 1. I have two services in my Keystone catalog: >> >> cinder: volume >> cinderv2: volumev2 >> >> It looks like you have 4: >> >> cinder: volume >> cinderv2: volumev2 >> cinder-hpc2n: volume >> cinderv2-hpc2n: volumev2 >> >> I don't know if that makes a difference, but maybe simplify the services >> to just the standard 2. >> >> 2. nova.conf on the nova-api server has the "os_region_name" setting set >> to the correct local region. I only had to restart the nova-api service to >> see changes take effect, but maybe restart nova-* -- who knows at this >> point! >> >> 3. Check the spelling of your regions. I noticed that you sometimes have >> "regionOne" and other times have "RegionOne" but maybe that is just a typo >> from sanitizing your data. >> >> Let me know if this helps :) >> >> Joe >> >> On Thu, Dec 10, 2015 at 2:31 AM, Salman Toor <salman.t...@it.uu.se> >> wrote: >> >>> Hi Joe, >>> >>> Thanks for your reply, yes Me and Marco are working together. >>> >>> I have tried to used >>> >>> os_region_name = RegionOne >>> >>> both in cinder.conf and nova.conf but no luck so far. But the error >>> message is a bit different. It seems like now the system is recognising the >>> region. >>> >>> >>> 2015-12-10 09:32:54.661 26792 TRACE nova.api.openstack EndpointNotFound: >>> internalURL endpoint for volume service named cinder in regionOne region >>> not found >>> >>> Following are my endpoints: >>> >>> —— >>> >>> *[root@controller: ~]* # openstack endpoint show cinder >>> >>> +--------------+------------------------------------------------+ >>> >>> | Field | Value | >>> >>> +--------------+------------------------------------------------+ >>> >>> | adminurl | http://cinder:8776/v1/%(tenant_id)s | >>> >>> | enabled | True | >>> >>> | id | e2693bcaf3da4be2810f04acd7995d7f | >>> >>> | internalurl | http://cinder:8776/v1/%(tenant_id)s | >>> >>> | publicurl | http://cinder:8776/v1/%(tenant_id)s | >>> >>> | region | regionOne | >>> >>> | service_id | 7bd5d667ec4b4d65b6c1b0de8b303fe3 | >>> >>> | service_name | cinder | >>> >>> | service_type | volume | >>> >>> +--------------+------------------------------------------------+ >>> >>> *[root@controller: ~]* # openstack endpoint show cinderv2 >>> >>> +--------------+------------------------------------------------+ >>> >>> | Field | Value | >>> >>> +--------------+------------------------------------------------+ >>> >>> | adminurl | http://cinder:8776/v2/%(tenant_id)s | >>> >>> | enabled | True | >>> >>> | id | 3000cb23c6ab4ee5b68876ee08257338 | >>> >>> | internalurl | http://cinder:8776/v2/%(tenant_id)s | >>> >>> | publicurl | http://cinder:8776/v2/%(tenant_id)s | >>> >>> | region | regionOne | >>> >>> | service_id | 61d8baeb4ee74c7798a60758b2f4171f | >>> >>> | service_name | cinderv2 | >>> >>> | service_type | volumev2 | >>> >>> +--------------+------------------------------------------------+ >>> >>> *[root@controller: ~]* # openstack endpoint show cinder-hpc2n >>> >>> +--------------+--------------------------------------------------+ >>> >>> | Field | Value | >>> >>> +--------------+--------------------------------------------------+ >>> >>> | adminurl | http://cinder:8776/v1/%(tenant_id)s | >>> >>> | enabled | True | >>> >>> | id | c088f1f9ca0248a08eb60d4ac8492150 | >>> >>> | internalurl | http://cinder:8776/v1/%(tenant_id)s | >>> >>> | publicurl | http://cinder:8776/v1/%(tenant_id)s | >>> >>> | region | HPC2N | >>> >>> | service_id | 473f3eab043f4130b30d9ee64fe43703 | >>> >>> | service_name | cinder-hpc2n | >>> >>> | service_type | volume | >>> >>> +--------------+--------------------------------------------------+ >>> >>> *[root@controller: ~]* # openstack endpoint show cinderv2-hpc2n >>> >>> +--------------+--------------------------------------------------+ >>> >>> | Field | Value | >>> >>> +--------------+--------------------------------------------------+ >>> >>> | adminurl | http://cinder:8776/v2/%(tenant_id)s | >>> >>> | enabled | True | >>> >>> | id | 985e5b8b068e4c3d99c12788236ee0fe | >>> >>> | internalurl | http://cinder:8776/v2/%(tenant_id)s | >>> >>> | publicurl | http://cinder:8776/v2/%(tenant_id)s | >>> >>> | region | HPC2N | >>> >>> | service_id | 089186fad4d7444b95e74ea9d7769423 | >>> >>> | service_name | cinderv2-hpc2n | >>> >>> | service_type | volumev2 | >>> >>> >>> +--------------+--------------------------------------------------+ >>> —— >>> >>> Again if I remove hpc2n everything started to work fine. Can you tell me >>> what service I suppose to restart (if required)? >>> >>> Regards.. >>> Salman >>> >>> >>> >>> PhD, Scientific Computing >>> Researcher, IT Department, >>> Uppsala University. >>> Senior Cloud Architect, >>> SNIC. >>> Cloud Application Expert, >>> UPPMAX. >>> Visiting Researcher, >>> Helsinki Institute of Physics (HIP). >>> salman.t...@it.uu.se >>> http://www.it.uu.se/katalog/salto690 >>> >>> On 09 Dec 2015, at 17:48, Joe Topjian <j...@topjian.net> wrote: >>> >>> Hi Salman, >>> >>> Someone mentioned this same issue yesterday in relation to Terraform >>> (maybe a colleague of yours?), so given the two occurrences, I thought I'd >>> look into this. >>> >>> I have a Liberty environment readily available, so I created a second >>> set of volume and volumev2 endpoints for a fictional region. Everything >>> worked as expected, so I started reviewing the config files and saw that >>> /etc/cinder/cinder.conf had an option >>> >>> [DEFAULT] >>> os_region_name = RegionOne >>> >>> I commented that out, but things still worked. >>> >>> Then in /etc/nova/nova.conf, I saw: >>> >>> [cinder] >>> os_region_name = RegionOne >>> >>> commenting this out caused volume attachments to hang indefinitely >>> because nova was trying to contact cinder at RegionTwo (I'm assuming this >>> is the first catalog entry that was returned). >>> >>> Given this is a Liberty environment, it's not accurately reproducing >>> your problem, but could you check and see if you have that option set in >>> nova.conf? >>> >>> I have a Kilo environment in the process of building. Once it has >>> finished, I'll see if I can reproduce your error there. >>> >>> Thanks, >>> Joe >>> >>> On Wed, Dec 9, 2015 at 4:35 AM, Salman Toor <salman.t...@it.uu.se> >>> wrote: >>> >>>> Hi, >>>> >>>> I am using Kilo release on CentOS. We have recently enabled multiple >>>> regions and it seems that Cinder have some problems with multiple >>>> endpoints. >>>> >>>> Thinks are working fine with nova but cinder is behaving strange. Here >>>> are my endpoints >>>> >>>> >>>> ———— >>>> *[root@controller: ~]* # openstack service list >>>> +----------------------------------+----------------+--------------+ >>>> | ID | Name | Type | >>>> +----------------------------------+----------------+--------------+ >>>> | 0a33e6f259794ff2a99e626be37c0c2b | cinderv2-hpc2n | volumev2 | >>>> | 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute | >>>> | 2c7828120c294d3f82e3a17835babb85 | neutron | network | >>>> | 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute | >>>> | 478eff4e96464ae8a958ba29f750b14c | glance | image | >>>> | 4a5a771d915e43c28e66538b8bc6e625 | cinder | volume | >>>> | 72d1be82b2e5478dbf0f3fb9e7ba969d | cinderv2 | volumev2 | >>>> | 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image | >>>> | a985795b49e2440db82970b81248c86e | cinder-hpc2n | volume | >>>> | dccd39b92ab547ddaf9047b38620145a | swift | object-store | >>>> | ebb1660d1d9746759a48de921521bfad | keystone | identity | >>>> +----------------------------------+----------------+--------------+ >>>> >>>> *[root@controller: ~]* # openstack endpoint >>>> show a985795b49e2440db82970b81248c86e >>>> +--------------+--------------------------------------------------+ >>>> | Field | Value | >>>> +--------------+--------------------------------------------------+ >>>> | adminurl | http://<URL>:8776/v1/%(tenant_id)s | >>>> | enabled | True | >>>> | id | d4003e91ddf24cfb9fa497da81b01a18 | >>>> | internalurl | http://<URL>:8776/v1/%(tenant_id)s | >>>> | publicurl | http://<URL>:8776/v1/%(tenant_id)s | >>>> | region | HPC2N | >>>> | service_id | a985795b49e2440db82970b81248c86e | >>>> | service_name | cinder-hpc2n | >>>> | service_type | volume | >>>> +--------------+--------------------------------------------------+ >>>> >>>> *[root@controller: ~]* # openstack endpoint >>>> show 4a5a771d915e43c28e66538b8bc6e625 >>>> +--------------+------------------------------------------------+ >>>> | Field | Value | >>>> +--------------+------------------------------------------------+ >>>> | adminurl | http://<URL>:8776/v1/%(tenant_id)s | >>>> | enabled | True | >>>> | id | 5f19c0b535674dbd9e318c7b6d61b3bc | >>>> | internalurl | http://<URL>:8776/v1/%(tenant_id)s | >>>> | publicurl | http://<URL>:8776/v1/%(tenant_id)s | >>>> | region | regionOne | >>>> | service_id | 4a5a771d915e43c28e66538b8bc6e625 | >>>> | service_name | cinder | >>>> | service_type | volume | >>>> +--------------+------------------------------------------------+ >>>> >>>> And same for v2 endpoints >>>> >>>> ———— >>>> >>>> ——— nova-api.log ——— >>>> >>>> achmentController object at 0x598a3d0>>, body: {"volumeAttachment": >>>> {"device": "", "volumeId": "93d96eab-e3fd-4131-9549-ed51e7299da2"}} >>>> _process_stack >>>> /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:780 >>>> 2015-12-08 12:52:05.847 3376 INFO >>>> nova.api.openstack.compute.contrib.volumes >>>> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1 >>>> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] Attach volume >>>> 93d96eab-e3fd-4131-9549-ed51e7299da2 to instance >>>> 3a4c8722-52a7-48f2-beb7-db8938698a0d at >>>> 2015-12-08 12:52:05.847 3376 DEBUG nova.compute.api >>>> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1 >>>> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] [instance: >>>> 3a4c8722-52a7-48f2-beb7-db8938698a0d] Fetching instance by UUID get >>>> /usr/lib/python2.7/site-packages/nova/compute/api.py:1911 >>>> 2015-12-08 12:52:05.988 3376 ERROR nova.api.openstack >>>> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1 >>>> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] Caught error: internalURL endpoint >>>> for volume service named cinder not found >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack Traceback (most >>>> recent call last): >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line >>>> 125, in __call__ >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> req.get_response(self.application) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack application, >>>> catch_exc_info=False) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in >>>> call_application >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack app_iter = >>>> application(self.environ, start_response) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> resp(environ, start_response) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", >>>> line 634, in __call__ >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> self._call_app(env, start_response) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", >>>> line 554, in _call_app >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> self._app(env, _fake_start_response) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> resp(environ, start_response) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> resp(environ, start_response) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in >>>> __call__ >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack response = >>>> self.app(environ, start_response) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> resp(environ, start_response) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack resp = >>>> self.call_func(req, *args, **self.kwargs) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> self.func(req, *args, **kwargs) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in >>>> __call__ >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack content_type, >>>> body, accept) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in >>>> _process_stack >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack action_result >>>> = self.dispatch(meth, request, action_args) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in >>>> dispatch >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> method(req=request, **action_args) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/volumes.py", >>>> line 305, in create >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume_id, >>>> device) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 230, in >>>> wrapped >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> func(self, context, target, *args, **kwargs) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 219, in inner >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> function(self, context, instance, *args, **kwargs) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 200, in inner >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> f(self, context, instance, *args, **kw) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3035, in >>>> attach_volume >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack disk_bus, >>>> device_type) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3016, in >>>> _attach_volume >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack >>>> volume_bdm.destroy() >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in >>>> __exit__ >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack >>>> six.reraise(self.type_, self.value, self.tb) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3009, in >>>> _attach_volume >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume = >>>> self.volume_api.get(context, volume_id) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in >>>> wrapper >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack res = >>>> method(self, ctx, volume_id, *args, **kwargs) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 286, in get >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack item = >>>> cinderclient(context).volumes.get(volume_id) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 115, in >>>> cinderclient >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack url = >>>> _SESSION.get_endpoint(auth, **service_parameters) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 635, in >>>> get_endpoint >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> auth.get_endpoint(self, **kwargs) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", >>>> line 219, in get_endpoint >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack >>>> service_name=service_name) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in >>>> inner >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return >>>> func(*args, **kwargs) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File >>>> "/usr/lib/python2.7/site-packages/keystoneclient/service_catalog.py", line >>>> 254, in url_for >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack raise >>>> exceptions.EndpointNotFound(msg) >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack EndpointNotFound: >>>> internalURL endpoint for volume service named cinder not found >>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack >>>> >>>> ——— >>>> >>>> If I remove cinder-hpc2n and cinderv2-hpc2n from the service list, >>>> everything started to work fine. >>>> >>>> This is how I have created endpoints: >>>> >>>> >>>> openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s >>>> --internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl >>>> http://<URL>:8776/v1/%\(tenant_id\)s >>>> --region regionOne 4a5a771d915e43c28e66538b8bc6e625 >>>> >>>> openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s >>>> --internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl >>>> http://<URL>:8776/v1/%\(tenant_id\)s >>>> --region HPC2N a985795b49e2440db82970b81248c86e >>>> >>>> using the service-id. >>>> >>>> >>>> Anyone experiencing this behavior? or Any suggestion how to fix it? >>>> Again regions are working fine with nova and glance. >>>> >>>> >>>> Regards.. >>>> Salman. >>>> >>>> >>>> PhD, Scientific Computing >>>> Researcher, IT Department, >>>> Uppsala University. >>>> Senior Cloud Architect, >>>> SNIC. >>>> Cloud Application Expert, >>>> UPPMAX. >>>> Visiting Researcher, >>>> Helsinki Institute of Physics (HIP). >>>> salman.t...@it.uu.se >>>> http://www.it.uu.se/katalog/salto690 >>>> >>>> >>>> _______________________________________________ >>>> OpenStack-operators mailing list >>>> OpenStack-operators@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >>>> >>>> >>> >>> >> >> >
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators