Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
On Fri, 2012-06-15 at 20:54 -0400, Lars Kellogg-Stedman wrote: Thanks for the reply, makes sense. Just to make sure I understand things, it sounds like Nova does not currently query Keystone for endpoints and continues to rely on explicit configuration (or to rephrase your answer, the reason these options have not gone away is because Nova does not yet have the necessary support for Keystone). Is that approximately correct? The problem with the Keystone endpoints is that you have to make a query to Keystone to get them. We want to reduce the number of hits we make on Keystone, not increase them—there are already too many as it is. Thus, I suspect that nova may not even use the Keystone endpoints. It *does* support image URLs, however. Thus, you use the options to configure the default glance endpoint, and if you want to hit another glance, you simply give a URL to the desired image rather than a simple identifier. (My comments about the support for endpoints in this email may differ from my previous comments; chalk that up to further reflection on the problem being solved…) -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
What's the point of a service catalog (list of endpoints) if we don't want to use it?! Looking up endpoints should be a cacheable request and in the grand scheme of things -- low impact. Nate On Jun 18, 2012 10:13 AM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Fri, 2012-06-15 at 20:54 -0400, Lars Kellogg-Stedman wrote: Thanks for the reply, makes sense. Just to make sure I understand things, it sounds like Nova does not currently query Keystone for endpoints and continues to rely on explicit configuration (or to rephrase your answer, the reason these options have not gone away is because Nova does not yet have the necessary support for Keystone). Is that approximately correct? The problem with the Keystone endpoints is that you have to make a query to Keystone to get them. We want to reduce the number of hits we make on Keystone, not increase them—there are already too many as it is. Thus, I suspect that nova may not even use the Keystone endpoints. It *does* support image URLs, however. Thus, you use the options to configure the default glance endpoint, and if you want to hit another glance, you simply give a URL to the desired image rather than a simple identifier. (My comments about the support for endpoints in this email may differ from my previous comments; chalk that up to further reflection on the problem being solved…) -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
On Mon, 2012-06-18 at 10:18 -0400, Nathanael Burton wrote: What's the point of a service catalog (list of endpoints) if we don't want to use it?! Looking up endpoints should be a cacheable request and in the grand scheme of things -- low impact. We do use the service catalog, quite extensively—on the client side. From nova to glance, I suspect we don't use the service catalog, since nova just uses the delegated credentials from the user. Looking up the service catalog is indeed quite cacheable; however: I don't believe that such code has been added; it may be necessary to pierce abstraction boundaries to perform that caching; and the glance endpoint is likely to be pretty static anyway, and thus fine for setting by means of configuration. And again, it has been a while since I looked at that code path… -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
I don't see nova-network running... And in fact, that seems to have been at the root of a number of problems. Thanks! With some work over the weekend I'm now successfully booting instances with networking using the Flat network manager. Great. It wasn't clear from the documentation that nova-network was a *necessary* service (that is, it wasn't clear that the failure mode would be fail to create an instance vs. your instance has not networking). We were punting on network configuration until after we were able to successfully boot instances...so that was apparently a bad idea on our part. -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
Thus, I suspect that nova may not even use the Keystone endpoints... That sounds crazy to me, but I just got here. That is, why go to the effort to develop an endpoint registration service and then decide not to use it? Given the asynchronous, distributed nature of OpenStack, an endpoint directory seems like a good idea. Just out of question, what *does* use the endpoint registry in KeyStone (in the Essex release)? -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
On Mon, 2012-06-18 at 10:41 -0400, Lars Kellogg-Stedman wrote: That sounds crazy to me, but I just got here. That is, why go to the effort to develop an endpoint registration service and then decide not to use it? Given the asynchronous, distributed nature of OpenStack, an endpoint directory seems like a good idea. Just out of question, what *does* use the endpoint registry in KeyStone (in the Essex release)? The clients. The endpoint registration system, so far as I understand, was primarily intended for use by the clients. It certainly would be useful for use by the servers, but there are subtleties, and I am not aware that it is currently used by nova-glance. But yet again, I have not looked at that code for a while; last time I was there, I was adding the initial support for nova to feed the user's credentials into glance; that was pre-Diablo, if I recall correctly. Nova, glance, keystone, etc. are all moving targets; there are tons of things that have only been added recently in the grand scheme of things, and there are many loose ends still to be tied. As an example, while I was rototilling the quotas system in nova, new quotas were added that changed the requirements I was working from, and since I was running up against deadlines, I had to leave some of those ends untied for now; there's no telling when I'll be able to get back to those loose ends and finally tie them up. I would not be surprised if something similar has happened WRT the endpoints system, since there are so many subtleties that need to be taken into account. -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
On 06/18/2012 10:41 AM, Lars Kellogg-Stedman wrote: Thus, I suspect that nova may not even use the Keystone endpoints... That sounds crazy to me, but I just got here. That is, why go to the effort to develop an endpoint registration service and then decide not to use it? Given the asynchronous, distributed nature of OpenStack, an endpoint directory seems like a good idea. It's mostly a vestigial thing. Before Keystone had an endpoint registry, we used configuration options to indicate important endpoint URLs. Just out of question, what *does* use the endpoint registry in KeyStone (in the Essex release)? I would imagine that most things will move towards using the endpoint registry over time and getting rid of multiple hardcoded endpoint URLs in configuration files. Best, -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] glance_api_servers vs. glance_host vs. keystone?
Howdy all, I've spent the past few days slogging through the initial steps of getting OpenStack and running and I seem to have hit a wall. If this isn't the right list for this question, please feel free to direct me elsewhere. I have two systems running the OpenStack components right now. The controller runs nova-api, nova-volume, nova-objectstore, glance, and keystone. The compute host runs nova-compute. All of the parts seemt to talk to each other successfully. For example, I can run 'nova image-list' on either system and get a list of available images: # nova image-list +--++++ | ID |Name| Status | Server | +--++++ | 383bbfab-01db-4089-a8fd-1a2735040af5 | DSL 4.4.10 | ACTIVE || +--++++ When I try to deploy a new guest using the 'nova boot' command: # nova boot --flavor m1.small --image 383bbfab-01db-4089-a8fd-1a2735040af5 lars0 The guest ends up permanently stuck in the BUILD state: # nova list +--+---++--+ | ID | Name | Status | Networks | +--+---++--+ | 06b343e6-bc6b-4e0b-baed-cda55cb85695 | lars0 | BUILD | | +--+---++--+ This is a surprisingly permanently condition. The server will never move out of the BUILD state, and it's not possible to delete this using 'nova delete', either. Looking at /var/log/nova/compute.log on the compute host, I don't see anything specific. I do see this: ERROR nova.rpc.impl_qpid [-] Timed out waiting for RPC response: None TRACE nova.rpc.impl_qpid Traceback (most recent call last): TRACE nova.rpc.impl_qpid File /usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py, line 359, in ensure TRACE nova.rpc.impl_qpid return method(*args, **kwargs) TRACE nova.rpc.impl_qpid File /usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py, line 408, in _consume TRACE nova.rpc.impl_qpid nxt_receiver = self.session.next_receiver(timeout=timeout) TRACE nova.rpc.impl_qpid File string, line 6, in next_receiver TRACE nova.rpc.impl_qpid File /usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py, line 651, in next_receiver TRACE nova.rpc.impl_qpid raise Empty TRACE nova.rpc.impl_qpid Empty: None On the controller, I'm seeing a lot of this in /var/log/nova/api.log: DEBUG nova.api.openstack.wsgi [req-5e4dc971-cb14-469a-9239-080a8c551b65 22bb8e502d3944ad953e72fc77879c2f 76e2726cacca4be0bde6d8840f88c136] Unrecognized Content-Type provided in request from (pid=1044) get_body /usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py:697 None of the command-line tools are producing any kind of visible error. I'm not really sure where I should be looking for problems at this point. Following is TMI: I'm running the Essex release under CentOS 6.2: # rpm -qa 'openstack*' openstack-utils-2012.1-1.el6.noarch openstack-glance-2012.1-5.el6.noarch openstack-dashboard-2012.1-4.el6.noarch openstack-quantum-2012.1-5.el6.noarch openstack-nova-2012.1-11.el6.noarch openstack-keystone-2012.1-3.el6.noarch openstack-swift-1.4.8-2.el6.noarch openstack-quantum-linuxbridge-2012.1-5.el6.noarch And the available endpoints are: +-++ | nova| Value | +-++ | adminURL| http://os-controller.int.seas.harvard.edu:8774/v2/76e2726cacca4be0bde6d8840f88c136 | | internalURL | http://os-controller.int.seas.harvard.edu:8774/v2/76e2726cacca4be0bde6d8840f88c136 | | publicURL | http://os-controller.int.seas.harvard.edu:8774/v2/76e2726cacca4be0bde6d8840f88c136 | | region | SEAS | | serviceName | nova | +-++ +-+---+ |glance | Value | +-+---+ | adminURL| http://os-controller.int.seas.harvard.edu:9292/v1 | | internalURL | http://os-controller.int.seas.harvard.edu:9292/v1 | | publicURL | http://os-controller.int.seas.harvard.edu:9292/v1 | | region | SEAS | +-+---+
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
Well, apologies for the stupid subject line. In the interests of sewing as much confusion as possible, here's the question that was supposed to go along with that subject: nova.conf appears to sport several configuration options related to glance, including: - glance_host - glance_port - glance_api_servers These seem suspiciously similar. Do they do the same thing? And shouldn't this information actually come from Keystone, in which there is an endpoint registered for the glance service? -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
Hey Lars, Sadly I don't have much in the way of a solution for you, but I do have some suggestions. Comments inline. On 6/15/12 4:17 PM, Lars Kellogg-Stedman l...@seas.harvard.edu wrote: Howdy all, I've spent the past few days slogging through the initial steps of getting OpenStack and running and I seem to have hit a wall. If this isn't the right list for this question, please feel free to direct me elsewhere. I have two systems running the OpenStack components right now. The controller runs nova-api, nova-volume, nova-objectstore, glance, and keystone. The compute host runs nova-compute. All of the parts seemt to talk to each other successfully. For example, I can run 'nova image-list' on either system and get a list of available images: # nova image-list +--++++ | ID |Name| Status | Server | +--++++ | 383bbfab-01db-4089-a8fd-1a2735040af5 | DSL 4.4.10 | ACTIVE || +--++++ When I try to deploy a new guest using the 'nova boot' command: # nova boot --flavor m1.small --image 383bbfab-01db-4089-a8fd-1a2735040af5 lars0 The guest ends up permanently stuck in the BUILD state: # nova list +--+---++--+ | ID | Name | Status | Networks | +--+---++--+ | 06b343e6-bc6b-4e0b-baed-cda55cb85695 | lars0 | BUILD | | +--+---++--+ This is a surprisingly permanently condition. The server will never move out of the BUILD state, and it's not possible to delete this using 'nova delete', either. Looking at /var/log/nova/compute.log on the compute host, I don't see anything specific. I do see this: ERROR nova.rpc.impl_qpid [-] Timed out waiting for RPC response: None TRACE nova.rpc.impl_qpid Traceback (most recent call last): TRACE nova.rpc.impl_qpid File /usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py, line 359, in ensure TRACE nova.rpc.impl_qpid return method(*args, **kwargs) TRACE nova.rpc.impl_qpid File /usr/lib/python2.6/site-packages/nova/rpc/impl_qpid.py, line 408, in _consume TRACE nova.rpc.impl_qpid nxt_receiver = self.session.next_receiver(timeout=timeout) TRACE nova.rpc.impl_qpid File string, line 6, in next_receiver TRACE nova.rpc.impl_qpid File /usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py, line 651, in next_receiver TRACE nova.rpc.impl_qpid raise Empty TRACE nova.rpc.impl_qpid Empty: None I'm used to using rabbit, but I did notice you didn't include a nova-scheduler in your list above, and this message seems to be saying it can't find an endpoint for qpidŠpossibly related? Again, I know nothing about qpid, but is there some way to see if the message is landing hitting qpid and getting stuck there? On the controller, I'm seeing a lot of this in /var/log/nova/api.log: DEBUG nova.api.openstack.wsgi [req-5e4dc971-cb14-469a-9239-080a8c551b65 22bb8e502d3944ad953e72fc77879c2f 76e2726cacca4be0bde6d8840f88c136] Unrecognized Content-Type provided in request from (pid=1044) get_body /usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py:697 We should probably figure out if anyone actually cares about this. It litters our logs, but seems to have no effect on anything. In any case you can ignore this error. One final piece of info that would be interesting to know is the vm_state and task_state from the db for the instances stuck in build. That would let us know just how far the instance got in the building process. My guess is that it is stuck in scheduling. Gabe None of the command-line tools are producing any kind of visible error. I'm not really sure where I should be looking for problems at this point. Following is TMI: I'm running the Essex release under CentOS 6.2: # rpm -qa 'openstack*' openstack-utils-2012.1-1.el6.noarch openstack-glance-2012.1-5.el6.noarch openstack-dashboard-2012.1-4.el6.noarch openstack-quantum-2012.1-5.el6.noarch openstack-nova-2012.1-11.el6.noarch openstack-keystone-2012.1-3.el6.noarch openstack-swift-1.4.8-2.el6.noarch openstack-quantum-linuxbridge-2012.1-5.el6.noarch And the available endpoints are: +-+--- -+ | nova| Value | +-+--- -+ | adminURL| http://os-controller.int.seas.harvard.edu:8774/v2/76e2726cacca4be0bde6d884 0f88c136 | | internalURL | http://os-controller.int.seas.harvard.edu:8774/v2/76e2726cacca4be0bde6d884 0f88c136 | | publicURL |
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
I'm used to using rabbit, but I did notice you didn't include a nova-scheduler in your list above... There is a nova-scheduler service running on the controller: Binary Host Zone Status State Updated_At nova-certos-controller.int.seas.harvard.edu nova enabled :-) 2012-06-15 20:41:44 nova-scheduler os-controller.int.seas.harvard.edu nova enabled :-) 2012-06-15 20:41:45 nova-volume os-controller.int.seas.harvard.edu nova enabled :-) 2012-06-15 20:41:47 nova-compute os-host.int.seas.harvard.edu nova enabled :-) 2012-06-15 20:41:46 That was just me accidentally omitting it from the list. can't find an endpoint for qpidŠpossibly related? Again, I know nothing about qpid, but is there some way to see if the message is landing hitting qpid and getting stuck there? I don't really know anything about AMQ or how OpenStack utilizes the message broker. Are there any commands I can run that would exercise the message broker and confirm whether or not it is working correctly? One final piece of info that would be interesting to know is the vm_state and task_state from the db for the instances stuck in build. That would let us know just how far the instance got in the building process. My guess is that it is stuck in scheduling. Well, scheduling and/or deleting, due to my failed attempt to delete some of these instances: mysql select id,hostname,vm_state,task_state from instances; ++--+--++ | id | hostname | vm_state | task_state | ++--+--++ | 1 | lars0| building | deleting | | 2 | lars1| building | scheduling | | 3 | lars0| building | deleting | | 4 | lars2| building | scheduling | ++--+--++ 4 rows in set (0.00 sec) -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
I don't really know anything about AMQ or how OpenStack utilizes the message broker. Are there any commands I can run that would exercise the message broker and confirm whether or not it is working correctly? For what it's worth: qpid provies a test suite (qpid-python-test that appears to complete successfully when run on the compute host using the broker on the controller. -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
Hi Lars - Sounds like you're really looking for a conceptual understanding and I like that! Keystone does maintain the service catalog, and the Image service (glance) needs specific entries. See http://docs.openstack.org/essex/openstack-compute/install/yum/content/keystone-concepts.html. I believe glance_host and glance_port are deprecated and not needed in an Essex installation, use only glance_api_servers in the nova.conf. Gabe has good answers for your previous questions - and hopefully you can reference this page for Qpid configuration information: http://docs.openstack.org/essex/openstack-compute/admin/content/configuration-qpid.html Anne On Fri, Jun 15, 2012 at 3:26 PM, Lars Kellogg-Stedman l...@seas.harvard.edu wrote: Well, apologies for the stupid subject line. In the interests of sewing as much confusion as possible, here's the question that was supposed to go along with that subject: nova.conf appears to sport several configuration options related to glance, including: - glance_host - glance_port - glance_api_servers These seem suspiciously similar. Do they do the same thing? And shouldn't this information actually come from Keystone, in which there is an endpoint registered for the glance service? -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist | http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
On Fri, 2012-06-15 at 16:26 -0400, Lars Kellogg-Stedman wrote: nova.conf appears to sport several configuration options related to glance, including: - glance_host - glance_port - glance_api_servers These seem suspiciously similar. Indeed. Do they do the same thing? Yes, they do. And shouldn't this information actually come from Keystone, in which there is an endpoint registered for the glance service? Yes, it should :) Now, a little history lesson: First came nova. Then, an index server was needed, and so the glance_host and glance_port options were added. Then, an enhancement: use of multiple glance hosts, and so glance_api_servers was added, with reasonable defaults drawn from glance_host and glance_port if it wasn't provided. Then, a centralized authentication service called Keystone was added, and as a benefit, it added the concept of endpoints. The reason these options have not gone away is probably a combination of supporting non-Keystone authentication and general programmer laziness… -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance_api_servers vs. glance_host vs. keystone?
The reason these options have not gone away is probably a combination of supporting non-Keystone authentication and general programmer laziness… Kevin, Thanks for the reply, makes sense. Just to make sure I understand things, it sounds like Nova does not currently query Keystone for endpoints and continues to rely on explicit configuration (or to rephrase your answer, the reason these options have not gone away is because Nova does not yet have the necessary support for Keystone). Is that approximately correct? -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
