[Openstack] openstack nova with thousands of hosts
Hi everyone, I want to understand the different possibilities to deploy Openstack Nova in a site with thousands of hosts. After reading the documentation and following the list for some time now I’m confused how to use the concepts: “region”, “availability zone”, “cell”, “host aggregate”… to have a good logical division. In the list was discussed that a “cell” (1 rabbit, 1 db, …) supports between 500-1000 hosts, so I understand that for a big site we need several “cells”. Cells can be nested in a tree fashion where each “child” can have several “parents”. Can I say that an availability zone is one of these trees of cells? Because a big site has heterogeneous hardware I can use the concept “aggregate host” for logically group similar resources. Aggregate hosts live inside cells. Are these assumptions correct? And how to deploy openstack nova between different sites (or regions…)? I saw some discussions for swift but none for nova. thanks, Belmiro Moreira ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] What does "nova migrate" do, exactly?
Hi, The host is selected by the scheduler. Belmiro On 05-06-2012 17:40, Florian Haas wrote: On 06/05/12 17:27, Vishvananda Ishaya wrote: No, migrate does "dead" migration on a server. Essentially snapshots the drive transfers it to a new server and reboots. Migrate and live-migration do need to be consolidated. Um, OK. How do I specify where to migrate to? nova help migrate usage: nova migrate [--poll] Migrate a server. Positional arguments: Name or ID of server. Optional arguments: --pollBlocks while instance migrates so progress can be reported. Slightly confused. :) Cheers, Florian ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Nova "migrate" and "resize"
Hi, I spent some time today trying understand how to use successfully the “resize” and “migrate” in nova. Because I couldn’t find any documentation… I share my experience. I have a kvm multi-node installation using fedora 16. Migrate and resize use ssh to create the directories and transfer the image. So you need to make sure that “nova” user is able to ssh the other compute nodes. Make sure you have this in your ssh configuration: Host * StrictHostKeyChecking no UserKnownHostsFile=/dev/null Create a ssh key for “nova” and distribute it in your setup. If you are using Fedora set selinux to permissive. … and should work. cheers, Belmiro ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova "migrate" and "resize"
Hi, I mention "nova" user because it's the default in the main distros. If I understood correctly your question, you can have different users... of course you need to distribute the right keys for ssh access and make sure they have the right permissions in your nova installation. cheers, Belmiro On Jun 6, 2012, at 2:12 AM, William Herry wrote: > Hi Belmiro, thanks for your share > > Is that required that the two node must have the same setting about the user, > like they all use nova run nova-* command, what if I set them different(one > of them run nova-* command with user 'stack', is it possible to make it work > also? > > Thanks > > On Wed, Jun 6, 2012 at 2:17 AM, Belmiro Moreira > wrote: > Hi, > I spent some time today trying understand how to use successfully the > “resize” and “migrate” in nova. Because I couldn’t find any documentation… I > share my experience. > I have a kvm multi-node installation using fedora 16. > > Migrate and resize use ssh to create the directories and transfer the image. > So you need to make sure that “nova” user is able to ssh the other compute > nodes. > > Make sure you have this in your ssh configuration: > Host * >StrictHostKeyChecking no >UserKnownHostsFile=/dev/null > > Create a ssh key for “nova” and distribute it in your setup. > If you are using Fedora set selinux to permissive. > … and should work. > > cheers, > Belmiro > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > > > -- > > > > William Herry > > williamherrych...@gmail.com > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova doesn't release ips when terminating instances
Hi, can you try with the flag: force_dhcp_release=false But first you need to: > Manually set instance_id=NULL in the fixed_ips table allows things to > work again. Belmiro CERN On Jun 23, 2012, at 5:11 AM, Lars Kellogg-Stedman wrote: > When an instance terminates, the "allocated" field in the fixed_ips is > set to "0", but the "instance_id" field remains set. Once all the > addresses are in this state, new instances fail to start, and the > following error is logged by nova-network: > > 2012-06-22 23:09:34 ERROR nova.rpc.amqp >[req-1fea207d-cd65-4375-9a04-17ba1ab92e3e >22bb8e502d3944ad953e72fc77879c2f 76e2726cacca4be0bde6d8840f88c136] >Returning exception Zero fixed ips available. to caller > > Which shows up in compute.log as: > > 2012-06-22 23:08:35 TRACE nova.rpc.amqp RemoteError: Remote error: > NoMoreFixedIps Zero fixed ips available. > > Manually set instance_id=NULL in the fixed_ips table allows things to > work again. > > We're running the 2012.1.1 release and we're using the FlatDHCP model. > Is this a known bug? > > Thanks, > > -- > Lars Kellogg-Stedman| > Senior Technologist| > http://ac.seas.harvard.edu/ > Academic Computing | > http://code.seas.harvard.edu/ > Harvard School of Engineering and Applied Sciences | > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] how to properly get rid of some `nova-manage service list` entries
Hi, you can simply delete them from the mysql services table. Belmiro CERN On Jul 9, 2012, at 12:46 PM, Christian Parpart wrote: > Hey all, > > I'm having some old entries in the output of `nova-manage service list`, > which I would like to get rid from. One compute, and 2 nova-network items. > Is it safe to just DELETE them from the mysql table or is there more involved? > > Best regards, > Christian Parpart. > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Image contextualization
Hello everyone, I'm interested in the functionallity described in the blueprint: https://blueprints.launchpad.net/nova/+spec/configuration-drive It was implemented for Diablo but I can't find any reference in the documentation. How to use this feature? Thanks a lot, Belmiro ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova/puppet blueprint, and some questions
Hi Andrew and others, first of all sorry for my previous email. (seems I sent a unfinished email) The idea of integrating nova with "configuration management tools" (puppet, chef, ...) is very interesting. But in my opinion we shouldn't have a specific implementation to a single tool or a small set of tools. In general Nova should facilitate the interaction with external tools (not only "configuration management tools"). In my view this could be achieved using: -> consistent contextualization method: The contextualization of new instances per project is crucial. We should have a "easy" way to transfer files into new instances early at boot time. I think documenting the blueprint could be a good start: https://blueprints.launchpad.net/nova/+spec/configuration-drive Because "injecting" files and metadata will always be image dependent. -> API extension: If nova API allowed the execution of "external scripts" when some operations (create, delete, ...) are performed the integration with external tools would be easier. --- What you have in your screenshot is really interesting but it seems very similar to other tools already implemented to manage puppet modules, ex: "Foreman". Belmiro Moreira CERN On Fri, Jan 27, 2012 at 2:56 AM, Andrew Bogott wrote: > Andrew -- > > Thanks for your comments. I'm going to start with a screenshot for context: > > http://bogott.net/misc/osmpuppet.png > > That's what it looks like when you configure an instance using Open Stack > Manager, which is WikiMedia's VM management interface. My main priority for > adding puppet support to Nova is to facilitate the creation and control of a > GUI much like that one. > > > On 1/26/12 5:03 PM, Andrew Clay Shafer wrote: > > > I'd also like to see more of a service oriented approach and avoid adding > tables to nova if possible. > > I'm not sure the best solution is to come up with a generic service for > $configuration_manager for nova core. I'd rather see these implemented as > optional first class extensions. > > This sounds intriguing, but I'll plead ignorance here; can you tell me more > about what this would look like, or direct me to an existing analogous > service? > > > What are you going to inject into the instances exactly? Where does the > site.pp live? > > This is the question I'm hoping to get feedback on. Either nova can > generate a fully-formed site.pp and inject that, or it can pass config > information as metadata, in which case an agent would need to be running on > the guest which would do the work of generating the site.pp. I certainly > prefer the former but I'm not yet clear on whether or not file injection is > widely supported. > > > I haven't thought about this that much yet, but off the top of my head, but > if the instances already have puppet clients and are configured for the > puppet master, then the only thing you should need to interact with is the > puppet master. > > > It's definitely the case that all of this could be done via LDAP or the > puppet master and involve no Nova action at all; that's how WikiMedia's > system works now. My aim is to consolidate the many ways we currently > interact with instances so that we delegate as much authority to Nova as > possible. That strikes me as generally worthwhile, but you're welcome to > disagree :) > > > I'm not a fan of the Available, Unavailable, Default, particularly because > you are managing state of something that may not be true on the puppet > master. > > I may be misunderstanding you, or my blueprint may be unclear. Available, > Unavailable, and Default don't refer to the availability of classes on the > puppet master; rather, they refer to whether or not a class is made > available to a nova user for a given instance. An 'available' class would > appear in the checklist in my screenshot. An Unavailable class would not. > A 'default' class would appear, and be pre-checked. In all three cases the > class is presumed to be present on the puppet master. > > > I also think managing a site.pp is going to be inferior to providing an > endpoint that can act as an eternal node tool for the puppet master. > http://docs.puppetlabs.com/guides/external_nodes.html > > In which case nova would interact directly with the puppet master for > configuration purposes? (I don't hate that idea, just asking for > clarification.) > > > > One other point, that you might have thought of, but I don't see anywhere on > the wiki is how to handle the ca/certs for the instances. > > I believe this (and your subsequent question) falls under the heading o
Re: [Openstack] Instances and ARP
Hi Joe, nova network filtering rules are preventing ip-spoofing. There is a proposal to modify this behavior when using HA in instances. See thread: [openstack-dev] VM level HA. Changes in firewall.py question. You can check with: virsh nwfilter-dumpxml nova-base cheers, Belmiro On Jan 21, 2013, at 12:25 PM, Joe Warren-Meeks wrote: > Hi guys, > > I've got openstack essex configured with vlanmanager and an external gateway > and all my networking runs ok generally. > > However, I'm trying to setup Linux HA on two instances. They run on separate > compute nodes and can see each other just fine. hb_takeover and hb_standby > works perfectly. The problem is that nothing outside of the instance with the > HA IP address can connect to it. > > It seems that something is ignoring the arp is-at from the instance. Doing a > tcpdump on the compute node's bridged network and the instance's eth0 I can > arp requests and responses fine for its main IP, but when I try to get to the > alias address, I see arp requests only on the compute side. On the instance > side I see it responding, but this doesn't show up on the bridged interface > on the compute node. > > Has anyone seen this before? My google-fu is failing to find anything. > > Kind regards > > -- joe. > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Ok, how do I debug this problem?
Hi Chris, Are all compute nodes happy? nova-manage service list Belmiro On Jan 21, 2013, at 7:23 PM, Christopher Smith wrote: > > So we have this four node Nova cluster running Essex. It has worked fine, > but we've added new network interfaces. Openstack has not yet been > configured to use them, but changes have been made that allows the system to > come up and configure the network in a similar way to before. Now, all of > the services seem to be up and running, Rabbit is apparently happy as far as > I can tell. Nova-scheduler and nova-compute are running and talking to > Rabbit. Now then, whenever I log into any nova node and boot an instance or > ten, all instances get started on a single node; always the same node, never > any attempt by any of the others to start an instance. I thought this would > be a Rabbit problem, but rabbit thinks it's doing fine on all nodes and the > cluster is intact. I've even tried completely rebuilding the Rabbit > configuration, just in case. If I stop services on the one working node, the > other nodes don't even appear to receive messages from either themselves or > each other, regardless of Rabbit's supposedly working state. > > Any ideas? Any clue how you would even test this to see what's happening? > > Chris > > -- > Christopher Smith > Systems Engineer, Wolfram Research > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] What scheduler should be used while using availability_zones
Hi, "SimpleScheduler" is deprecated in Folsom and it only covers volume scheduling. In Folsom use the "FilterSchedule" and make sure you have the filter AvailabilityZones enabled. Belmiro On Jan 25, 2013, at 7:52 AM, Balamurugan V G wrote: > Hi, > > I have two compute nodes in my setup each one is separate zones. I am using > the following setting in the nova.conf. > > scheduler_driver=nova.scheduler.simple.SimpleScheduler > > But when I launch instance and specify a zone, i still see that some times it > creates the instance in the wrong zone. Could this be due to the scheduler? > What scheduler should I use for this? > > I want a deterministic way to create instances in specific zones. > > Thanks, > Balu > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] How to resize an instance if the image was deleted from glance?
Hi, I'm trying to resize an instance but it fails with: ERROR: The resource could not be found. (HTTP 404) (Request-ID: req-37976706-cb02-46df-b7d8-11ceb96da63f) This is because nova expects the image is in glance to do the migration/resize. If an image is removed from glance we can't resize/migrate the instances that are using that image. In this scenario can we consider copy not only the "delta" but also the base image to the new compute node? Or convert it to raw and copy this image file? thanks, Belmiro ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova scheduler: AggregateMultiTenancyIsolation enhanced for domain level tenancy
Hi, I never thought about domains but it seems a great idea! If you want to extend it would be great. Remember that at the moment the "AggregateMultiTenancyIsolation" filter only guaranties that the nodes in the aggregate will run instances from the specified tenant. But it isn't exclusive… There is a blueprint to extend it: https://blueprints.launchpad.net/nova/+spec/multi-tenancy-isolation-only-aggregates At CERN we already implemented this blueprint and we also support of multiple tenants in the same aggregate. We use this filter to have dedicated resources for some tenants. I didn't upload the code yet… in my TODO list but also waiting for the blueprint approval. cheers, Belmiro On May 3, 2013, at 8:21 PM, boden wrote: > Hi, > I was poking around for a way to scope host aggregates at the domain level > and moreover support the notion multiple tenant access. > > For example: > * As an Cloud provider, I want to isolate aggregates/availability zones at > the domain level (i.e. assign which domain can spawn instances on a given > aggregate/az) > * As an Cloud provider, I want to isolate aggregates/availability zones at > the domain level and permit multiple domains or projects access (i.e. same as > previous UC but permit a set of domains access to the aggregate/az) > > I came across the AggregateMultiTenancyIsolation scheduler filter and it > appears it might be fairly simple to enhance this filter to: > (a) support domains in addition to tenants/project (i.e. filter_domain_id in > aggregate metadata) > (b) support multiple domains/projects access (i.e. allow a comma list or > array or domains/tenants on the aggregate metadata) > > Anyone else see a need for this? > > Disclaimer: nova n00b here, so my above assertions may be off target. > > Thx > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] more info on this raise exception.NoValidHost
Hi, Let me just had that an instance can fail in different filters in different hosts. Some filters LOG the reason if debug=true. Check /var/log/nova/scheduler.log cheers, Belmiro On May 21, 2013, at 3:18 PM, Lei Zhang wrote: > Yes, nova-scheduler has lots of filters to check if nova-compute node fit the > requirement of the instance you wanna launch. > Like availability-zone filter, RAM filter, cpu core filter, etc. If all the > compute node failed to pass those filters, it raise this exception. > > Regards > Lei > > > On Tue, May 21, 2013 at 4:13 AM, Aaron Rosen wrote: > Usually means that the instance trying to be launched can't be scheduled to > any nova-compute nodes. The log message should include a reason why though. > > > > > On Mon, May 20, 2013 at 12:06 PM, Remo Mattei wrote: > Hello everyone, > > I wonder if someone can give me a more detail overview about the raise > exception.NoValidHost > on what does raise mean. > > Thanks > Remo > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Glance client with API 2
Hi, reading the docs my understanding is that if we use glance API v2 the glance-registry is not needed. I tried to remove glance-registry from my setup and use api v2 but then the glance client doesn't offer the same functionality. How to create an image using glance client and api v2? thanks, Belmiro ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance client with API 2
Hi Iccha, thanks for the Blueprint. cheers, Belmiro On May 28, 2013, at 7:23 PM, Iccha Sethi wrote: > Hey Belmiro, > > You are right that python-glanceclient does not support v2 api entirely yet > and supports only limited actions. But we are working towards it supporting > all the operations. Here is the blue print for it : > https://blueprints.launchpad.net/python-glanceclient/+spec/glance-client-v2/ . > > But if you are interested, and want to experiment with v2 api, we will be > happy to provide the curl reqs to do so. > > Thanks, > Iccha > > > -Original Message- > From: "Belmiro Moreira" > Sent: Tuesday, May 28, 2013 9:08am > To: "openstack@lists.launchpad.net" > Subject: [Openstack] Glance client with API 2 > > Hi, > reading the docs my understanding is that if we use glance API v2 the > glance-registry > is not needed. > I tried to remove glance-registry from my setup and use api v2 but then the > glance client > doesn't offer the same functionality. > > How to create an image using glance client and api v2? > > thanks, > Belmiro > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] some issues about devstack
Hi, devstack is not for production environments... But answering your questions, you can change nova.conf as you like and then unstack and rejoin-stack Belmiro On 22 Jul 2013, at 04:25, 姚牧阳 wrote: > Hi, all > > I have been using openstack for quite a long time and recently I want to > deploy an experimental environment on 8 machines using devstack. Everything > seems to work fine but there is still some issues that disturbs me > > (1) nova logs: > In a productive deployment, as I did before, I can freely read the logs from > "/var/log/nova". However, it seems that in devstack, I can only use the > SYSLOG configuration to gather the logs. However, the logs of 8 machine is > too large to handle and I want to read specificly the nova-scheduler logs. > How can I make it log to "/var/log/nova" as before? > > (2) compute-scheduler choices > What I really want to do in nova is to add some new features by modifying the > compute-scheduler. So I need to modify "compute_scheduler_driver" in > "/etc/nova/nova.conf" and use a new scheduler class. But how can I do this > via devstack? I tried to add "SCHEDULER=~~" in "devstack/localrc" but it > didn't work. Anyone have any idea on that? > > (3) use the modified /etc/nova/nova.conf > After I changes some configurations in nova.conf, the services need to be > restarted to use the configs. But the only way to do so (as far as I'm > concerned) is to ./unstack.sh + ./stack.sh, which will overwrite all the > previous changes in "nova.conf". Is there any way that I can stop a certain > service and then start it over to use the new configurations. > > thanks > > Eric Yao > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
