Re: [Openstack] Grizzly Bear
As long as the network the interfaces connect to is properly configured for the traffic you are trying to put across them there's no reason I know of that OpenStack should care if they are physical or virtual (or tunnels or bonded interfaces, or what). That said I've not tried virtual interfaces. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Checkpoint VM and roll back
On Thu, Apr 25, 2013 at 2:06 PM, Tim Bell tim.b...@cern.ch wrote: I'd like to be able to 1. checkpoint a running virtual machine 2. run a test 3. rollback to the checkpoint from step 1 Has anyone had experience of doing this using OpenStack (such as with snapshots) ? For slow cycling tests like revising our base operating system images or deploying large new config management blocks I: 1) boot from a base image 2) deploy and test change 3) on pass snapshot on fail destroy and relaunch last good snapshot 4) recurse as required for multi layered change tests For faster cycles OpenStack snapshoting may be too cumbersome. I've used LVM and ZFS snapshots to provide in operating system roll back and have read of others doing CoW overlays (unionfs as similar) but that's operating system specific on openstack. -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] multi-host mode in quantum
Hi All, On Thu, Apr 4, 2013 at 3:23 PM, Henry Gessau ges...@cisco.com wrote: Unfortunately, I don't think multiple network nodes is the same multi-host network mode that Xin is asking about. The following did not make it into grizzly and is now targeted for havana: https://blueprints.launchpad.net/quantum/+spec/quantum-multihosthttps://blueprints.launchpad.net/quantum/+spec/quantum-multihost I'm hoping that means there still needs to be a central dhcp server but the multiple L3 agents mean actual traffic from VMs can be managed directly on the compute note they are running on and doesn't need to be sent back to a different network node before being nat'ed or tagged? Multi-host was the single most important feature for me in the quantum blue prints, if nat'ed networks need to be piped through a gateway box other than the physical host the instance is on quantum remains a no go for me. Having a central (but redundant) dhcp with distributed NAT may actually be an improvement over having to run dnsmasq literally everywhere. -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Network layout/planning for HA
On Fri, Mar 22, 2013 at 03:25:56PM -0400, Samuel Winchenbach wrote: :It looks like 802.3ad is for making parallel, redundant connections. I :only have two NICs in the server; I don't think I would want to bind by :internal and external NICs together would I? What you want is up to you :) You could though if you then use VLANs to segregate the internal and external traffic over the bonded link. We're not doing this in our OpenStack but do have older production virtulization systems where we're using VLANs over bonded NICs. -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Help with VMs
On Tue, Feb 19, 2013 at 1:25 PM, Harvey West harvey.w...@btinternet.com wrote: This boots ok. kvm -m 2048 -hda freeBSD.img -boot c (note: did not use virtio mods. Assumed these were just optimized NIC/SCSI drivers. Which I can live without for the time being) I ran into the same isssue with FreeBSD images. These are paravirtualized drivers which are absolutely required to get the VM to work. Linux images usually have them already but for FreeBSD (and Windows) you need to install them by hand. note that when using kvm by hand you can specify the driver emulation required so this is why it boots by hand but not via openstack (you could also configure openstack to use different kvm settings, but the virtio driver is faster and installing the drivers on FreeBSD isn't too hard. I'm not much of a *BSD guy but I managed) -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Help, erroneous resource tracker preventing instances from starting
Hi All,
I have a growing problem in which compute nodes are puzzlingly over
reporting their resource utilization and thus appearing to be over utilized
when they are in fact empty. System is Ubuntu 12.04 using cloud archive
Folsom (2012.2-0ubuntu5~cloud0) problem appeared on a single node after
upgrade from Essex some months ago and has now grown to 5 nodes (the lowest
numbered 5 nodes both by IP and lexically by name)
For example on the compute node nova-1:
2013-01-07 10:39:43 INFO nova.compute.manager [-] Updating host status
2013-01-07 10:41:02 AUDIT nova.compute.resource_tracker [-] Free ram (MB):
-397134
2013-01-07 10:41:02 AUDIT nova.compute.resource_tracker [-] Free disk (GB):
-3318
2013-01-07 10:41:02 AUDIT nova.compute.resource_tracker [-] Free VCPUS: -215
2013-01-07 10:41:02 INFO nova.compute.resource_tracker [-] Compute_service
record updated for nova-1
Oddly even though no instances are scheduled teh resource utilization does
vary, for example in the last 5hours:
root@nova-1:~# grep 'Free VCPUS:' /var/log/nova/nova-compute.log|awk
'{print $NF}'|sort -n |uniq -c
156 -218
3 -216
5 -215
2 -214
2 -212
1 -211
1 -210
5 -209
5 -208
# but no instances are running
root@nova-1:~# virsh list
IdName State
root@nova-1:~#
# nor does OpenStack seem to *think* any instances are running or reserved
by any projects
# as seen by nova-manage service describe_resource nova-1
HOST PROJECT cpu mem(mb) hdd
nova-1 (total)24 48295 602
nova-1 (used_now)233 4331413740
nova-1 (used_max) 0 0 0
# note lack of a list of tenants here
I can't replicate the issue intetionally but also can't clear appaerent
resource utilization. Tried direct manipulation of the database but that
gets reset by computenode reports, tried rebooting the nodes. I can always
fall back to just reinstalling them, but since this is still a
pre-production cluster I'd liek to understand what is happening.
Anyone have an insight into why nova.compute.resource_tracker is so
confused or how I can force it to understand what resources are in use?
Operationally it isn't painful to reinstall, but it does hurt a bit not
knowing what's going on here.
Thanks,
-Jon
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] How to deploy openstack automatically in your env.
On Mon, Dec 03, 2012 at 06:21:54PM +0800, Lei Zhang wrote: :It is a wired thing that the openstack is a python project. But many tools :for it are build on ruby? Puppet (http://puppetlabs.com/) and Chef (http://www.opscode.com/chef), the main players in configuration management are both written in Ruby and manage more than just OpenStack so the code just isn't related I can't be much help on your origina RHEL question as I run a Debian / Ubuntu shop. But while it is true that deployment tools are better tested on Ubuntu, certainly any puppe tor chef based solutions *should* work on RHEL if they don't it would be worth reporting their failures to the developers of those tools. There is a Fedora OpenStack wiki at http://fedoraproject.org/wiki/OpenStack which is linked from http://www.openstack.org/software/start/ which has other RedHat family specific links at the bottom of the page -Jon : : :On Mon, Dec 3, 2012 at 5:44 PM, Joe Breu joseph.b...@rackspace.com wrote: : : Hi Lei, : : We have chef cookbooks to install Openstack located at : http://github.com/rcbops/chef-cookbooks. : : --- : Joseph Breu : Deployment Engineer : Rackspace Private Cloud : 210-312-3508 : : On Dec 3, 2012, at 9:15 AM, Lei Zhang wrote: : : Hi all, : : I search the internet for days and found several automatically tool. : Including : :- devstack :- puppet+pupet-openstack :- stackops :- OneStack : : But It seems that all the scripts are well tested on ubuntu not RHEL. How : could you guys to deploy the openstack automatically, especially on RHEL. : -- : Lei Zhang : : Blog: http://jeffrey4l.github.com : twitter/weibo: @jeffrey4l : : ___ : Mailing list: https://launchpad.net/~openstack : Post to : openstack@lists.launchpad.net : Unsubscribe : https://launchpad.net/~openstack : More help : https://help.launchpad.net/ListHelp : : : : : :-- :Lei Zhang : :Blog: http://jeffrey4l.github.com :twitter/weibo: @jeffrey4l :___ :Mailing list: https://launchpad.net/~openstack :Post to : openstack@lists.launchpad.net :Unsubscribe : https://launchpad.net/~openstack :More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [Nova] used_max -vs used_now resources?
Hi All, I'm trying to make sense of 'nova-manage service describe_resource' output from Folsom running on Ubuntu 12.04 with KVM back end. I'd thought used_now accounted for running resource use and used_max included reservations that had not yet been instantiated. That assumption seemed to match what I saw under Essex. now I have situations like this: HOST PROJECT cpu mem(mb) hdd nova-3 (total)24 48295 605 nova-3 (used_now)140 2872323400 nova-3 (used_max) 48192 90 nova-3 956ae20bbb444a8c8f149729198aec63 4 8192 90 where used_now is wildly higher than it seems it should be. looking on the compute node itself verifies there are 4 single vCPU instances running there. On a different system reporting a high used_now but zero actually running instances I tried rebooting the compute node hoping to clear state (since the node pushes these numbers to the database correcting them there doesn't work). On reboot it came up claiming over 160 cpus in use. Is this a bug, a misconfiguration, or a misunderstanging? Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] How to I unconfuse my resource_tracker?
My compute nodes are confused about how many resource they have free. I suspect this is largely due to RPC timeouts I was experiencing due to a misconfiguration compounded by high load and a scheduler bug, but not so much interested in how it got this way as to how to clean it up. for example on node nova-1 the are actually 7 single CPU instances running as shown by virsh on the system and in the instances table and 24 available vCPUs. However it reports -102 VCPUs available: root@nova-1:~# grep AUDIT /var/log/nova/nova-compute.log|tail -4 2012-11-06 15:49:20 AUDIT nova.compute.resource_tracker [-] Free VCPUS: -102 2012-11-06 15:50:50 AUDIT nova.compute.resource_tracker [-] Free ram (MB): -205145 2012-11-06 15:50:50 AUDIT nova.compute.resource_tracker [-] Free disk (GB): -2175 2012-11-06 15:50:50 AUDIT nova.compute.resource_tracker [-] Free VCPUS: -102 nova-manage service describe_resource nova-1 HOST PROJECT cpu mem(mb) hdd nova-1 (total)24 48295 605 nova-1 (used_now)126 2534402780 nova-1 (used_max) 7 14336 210 nova-1 3008a142e9524f7295b06ea811908f93 7 14336 210 Both of these match what I see in the compute_nodes table which also tells tells me there are 88 running_vms (not 7). Where is the resource_tracker on nova-1 getting this information and how do I correct it? (It's clearly pushing this bad info back to the database as I first tried to correct it there) -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Scheduler issues in folsom
On Wed, Oct 31, 2012 at 10:54 PM, Vishvananda Ishaya vishvana...@gmail.comwrote: My patch here seems to fix the issue in the one scheduler case: https://github.com/vishvananda/nova/commit/2eaf796e60bd35319fe6add6dd04359546a21682 If you could give that a try on your scheduler node and see if it fixes it that would be awesome. Also, it would be fery helpful if you can report a bug for me to reference in my merge proposal. I will see what I can do to write a few tests and have a potential fix for multiple schedulers. bug is here https://bugs.launchpad.net/nova/+bug/1073956 since you've reproduced it If give it a try == drop that host_manager.py in place of my (folsom) file and restart the scheduler I'm still getting the same results when using a 100 iteration for loop around nova boot --availability-zone az:host all 100 end up on nova-1 but I'm suspicious this may avoid the scheduler entirely not sure how that availability zone trick for specifying a target hso tis implemented. My user's case that I'm trying to make work uses 'euca-run-instances -n 500' using that with a value of 200 (and your host_manager.py) the scheduler immediately puts them all in error state and doesn't say any more about it, previously it was scheduling them poorly. It's quite possible I've knocked something loose while banging around (also possible I need to pull your whole branch, didn''t look as closely as I should have at what it was based on) so I'm going to recheck my services and do some more tests but that's what I see at first. Thanks! -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Scheduler issues in folsom
Hi All While the RetryScheduler may not have been designed specifically to fix this issue https://bugs.launchpad.net/nova/+bug/1011852 suggests that it is meant to fix it, well if it is a scheduler race condition which is my suspicion. This is my current scheduler config which gives the failure mode I describe: scheduler_available_filters=nova.scheduler.filters.standard_filters scheduler_default_filters=AvailabilityZoneFilter,RamFilter,CoreFilter,ComputeFilte r,RetryFilter scheduler_max_attempts=30 least_cost_functions=nova.scheduler.least_cost.compute_fill_first_cost_fn compute_fill_first_cost_fn_weight=1.0 cpu_allocation_ratio=1.0 ram_allocation_ratio=1.0 I'm running the scheduler and api server on a single controller host and it's pretty consistent about scheduling hundred instances per node at first then iteratively rescheduling them elsewhere when presented with either an single API request to start many instances (using euca2ools) or a shell loop around nova boot to generate one api request per server. the cpu_allocation ratio should limit the scheduler to 24 instances per compute node regardless how how it's calculating memory, so while I talked a lot about memory allocation as a motivation it is more frequent for cpu to actually be the limiting factor in my deployment and it certainly should. And yet after attempting to launch 200 m1.tiny instances: root@nimbus-0:~# nova-manage service describe_resource nova-23 2012-10-31 11:17:56 HOST PROJECT cpu mem(mb) hdd nova-23 (total)24 48295 882 nova-23 (used_now)107 56832 30 nova-23 (used_max)107 56320 30 nova-23 98333a1a28e746fa8c629c83a818ad57 106 54272 0 nova-23 3008a142e9524f7295b06ea811908f93 1 2048 30 eventually those bleed off to other systems though not entirely 2012-10-31 11:29:41 HOST PROJECT cpu mem(mb) hdd nova-23 (total)24 48295 882 nova-23 (used_now) 43 24064 30 nova-23 (used_max) 43 23552 30 nova-23 98333a1a28e746fa8c629c83a818ad57 42 21504 0 nova-23 3008a142e9524f7295b06ea811908f93 1 2048 30 at this point 12min later out of 200 instances 168 are active 22 are errored and 10 are still building. Notably only 23 actual VMs are running on nova-23: root@nova-23:~# virsh list|grep instance |wc -l 23 So that's what I see perhaps my assumptions about why I'm seeing it are incorrect Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Scheduler issues in folsom
On Wed, Oct 31, 2012 at 1:47 PM, Huang Zhiteng winsto...@gmail.com wrote: Hi Jonathan, If I understand correctly, that bug is about multiple scheduler There is only a single process, I was reading it as relating to include threads within a single process, but they should clearly be able to serialize this withing a single process, so hoping my reading is not correct. What openstack package are you using? 2012.2-0ubuntu5~cloud0 on Ubuntu 12.04 from Ubuntu cloud-archive. With further testing it appears the allocation ratios are simply being ignored as the ChanceScheduler also violates my resource allocation limits I just hadn't been testing at large enough scale to reliably see it. I am seeing memory allocations 1.5 default ram_allocation_ratio which is somewhat curious. So it's almost as if the RamFilter isn't filtering rather than falling back to the default allocation ration (the compute nodes would melt before I got to the 16:1 default cpu allocation ration so not surprised I'm not getting near that more like 2:1) again despite: scheduler_available_filters=nova.scheduler.filters.standard_filters scheduler_default_filters=AvailabilityZoneFilter,RamFilter,CoreFilter,ComputeFilter,RetryFilter cpu_allocation_ratio=1.0 ram_allocation_ratio=1.0 Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Scheduler issues in folsom
Hi All, I'm having what I consider serious issues with teh scheduler in Folsom. It seems to relate to the introdution of threading in the scheduler. For a number of local reason we prefer to have instances start on the compute node with the least amount of free RAM that is still enough to satisfy the request which is the reverse of the default policy of scheduling on the system with the most free RAM. I'm fairly certain the smae behavior would be seen with that policy as well, and any other policy that results in a best choice for scheduling the next instance. We have work loads that start hundreds of instances or the same image and there are plans on scaling this to thousands. What I'm seeing is somehting like this: * user submits API request for 300 instances * scheduler puts them all on one node * retry schedule kicks in at some point for the 276 that don't fit * those 276 are all scheduled on the next best node * retry cycle repeats with the 252 that don't fit there I'm not clear exactly where the RetryScheduler in serts itself (I should probably read it) but the first compute node is very overloaded handling start up request which results in a fair number of instances entering ERROR state rather than rescheduling (so not all 276 actually make it to the next round) and the whole process it painfully slow. In the end we are lucky to see 50% of the requested instances actually make it into Active state (and then only becasue we increased scheduler_max_attempts). Is that really how it's supposed to work? With the introduction of the RetryScheduler as a fix for the scheduling race condition I think it is, but it is a pretty bad solution for me, unless I'm missing something, am I? wouln't be the first time... For now I'm working around this by using the ChanceScheduler (compute_scheduler_driver=nova.scheduler.chance.ChanceScheduler) so the scheduler threads don't pick a best node. This is orders of magnitude faster and consistantly successful in my tests. It is not ideal for us as we have a small minority of ciompute nodes with twice the memory capacity of our standard nodes and would prefer to keep those available for some of our extra large memory flavors and we'd also liek to minimize memory fragmentation on the standard sized nodes for similar reasons. -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Special Zero Size root in Flavor, what exactly does it mean?
Hi All, I know that specifying a zero size root volume in a flavor is special but what exactly does it mean. I'm writing up some documentation on flavors and realized I can't fully explain the default m1.tiny ... Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Using nova-volumes openstack LVM group for other pourposes
On Thu, Oct 25, 2012 at 12:48:23PM +0200, Daniel Vázquez wrote: :If we create a volume out of openstack context (it's is not created on :nova database) only on system level. Can we after add to openstack? :reformulate ask, Can we add to openstack an existen volumen (it's :living on nova-volumes group)? I'm not sure on nova-volume create :command or horizon ... that behavior it will be with existent lv label :or some other properties. It would be possible, it would require manually inserting data in many places though...this definately isn't a supported option but I do belive it is technically possible you might need to follow the naming convention (probably should) of volume-uuid, this could be doen with lvrename you will need to make an entry in the volumes table of your nova database you will also need an entry in /var/lib/nova/volumes/volume-uuid so that tgtd knows how to map the iSCSI target back to the correct device. that's what comes to mind, it's untested and there may be other pieces, but there isn't anything special in the logical volume data or metadata. If you're trying to plot a migration path for existing VMs it might be better to use nova to create a new volume and 'dd' to dump the contents of the old volume into it just to be sure everthing lines up correctly. -Jon : :2012/10/24 Daniel Vázquez daniel2d2...@gmail.com: : Yeah!! Jon, I'm agree with you about organization/separation LVM : groups, this is for very very very special situation. : Any case if I use nova pattern labeling via logical volumen creation : or via renaming label, I hope can switch the content of this custom : logical volument to use with openstack, an attach to a VM in future. : : : : 2012/10/24 Jonathan Proulx j...@csail.mit.edu: : On Wed, Oct 24, 2012 at 08:56:26PM +0200, Daniel Vázquez wrote: : :Hi here! : : : :Can we create and use news logical volumes for own/custom use(out of : :openstack) on nova-volumes openstack LVM group, and use it beside : :openstack operational? : :IMO it's LVM and no problem, but it has openstack collateral consequences? : : If you are talking about creating random logical volumes for : non-openstack use in the same volume group nova-volume or cinder is : using to create volumes (lv are in the same vg but don't otherwise : interact), yes you can do that without ocnfusing openstack or having : your volumes trampled. For example only having one volume group and : using that for operating system partitions as well as volume-foo : volumes for cinder : : I don't think it's a particularly good idea from an organizational : standpoint I'd rather have distinct vg's for each purpose so it is : clear which resources are operating system and which are data, but in : my environment (a private computing/research cloud with a small admin : group and 1k users in a few 10's of closely related tenents) it's : probably more an aesthetic than technical choice. The larger and more : diverse your situation the stronger I'd argue for keeping them in : seperate VGs. : : -Jon : :___ :Mailing list: https://launchpad.net/~openstack :Post to : openstack@lists.launchpad.net :Unsubscribe : https://launchpad.net/~openstack :More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Possible upgrade bug in nova-volume ( cinder)?
On Wed, Oct 24, 2012 at 6:32 PM, John Griffith john.griff...@solidfire.com wrote: Hey Jon, Cool... Yeah, I had intended for that patch to be a stable/folsom patch but shouldn't have submitted it to master :( The real problem isn't just normalizing the lvm names, but also the provider_location information that is stored in the DB for when you try to attach to your compute node. Yup, I had understood that. We only have about 10 active volumes so it was easy enough in my case to copy and paste the ids/uuids around for mysql updates and lvrenames. So I can now attach both old and new volumes at my site. The cinder version is the same (https://review.openstack.org/#/c/14790/) and depending on the feedback it's a candidate for back-port. The other possible issues that I've seen people run in to: 1. volumes directory not being specified correctly 2. not restarting tgtadm 3. not having the include statement in /etc/tgt/conf.d I think you and I covered most of these in our chat on IRC earlier today... I tripped over a couple more steps that I didn't know about but managed to discover on my own as well, like 'cinder-manage db sync' to set up the tables in the cinder database, which I'm sure are so obvious as to be nearly invisible to people working on the code and enough like any other openstack service it wasn't too hard to guess. I collected up my notes and expanded migration section of the release notes by a few steps http://wiki.openstack.org/ReleaseNotes/Folsom#OpenStack_Block_Storage_.28Cinder.29 Thanks again, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Possible upgrade bug in nova-volume ( cinder)?
Hi All, I'm seeing a bug due to my recent essex to folsom upgrade relating to LVM back volume storage, I'm not sure where it got introduced most likely either in nova-volume or in the Ubuntu cloud archive packaging...I only noticed it after transitioning from folsom-nova-volume to fosom-cinder but despite thinking I'd tested the nova-volume service before moving to cinder I'm pretty sure it had to exist in nova-volume as well (prehaps it was masked becasue I didn't restart tgtd until cinder) The symptom is that volumes created under folsom (with nova-volume or cinder) can be attached. The reason is that the backing-store devices in both /var/lib/nova/volumes/* and /var/lib/cinder/volumes/* are all named /dev/vg/volume-uuid while under essex the volumes were named /dev/vg/volume-id To fix this for me I can look up the volumes by ID in the database and then lvrename the logical volumes (I don't have too many and all on one volume server right now). Before I go sifting through postinst scripts and openstack code to see whre this came from anyone know where I should file this (and has anyone else run into it)? -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Possible upgrade bug in nova-volume ( cinder)?
On Wed, Oct 24, 2012 at 2:45 PM, Jonathan Proulx j...@jonproulx.com wrote: To fix this for me I can look up the volumes by ID in the database and then lvrename the logical volumes (I don't have too many and all on one volume server right now). That maybe the wrong answer as the database (both cinder and the older nova leavings) has a provider_location that implies the right logical volume name: +--++---+ | id | ec2_id | provider_location | +--++---+ | 25cb6abc-1938-41da-b4a4-7639fa122117 | NULL | 128.52.x.x:3260,9 iqn.2010-10.org.openstack:volume-001c 1 | | 60cd2c0e-6d61-4010-aee2-df738adb3581 | NULL | 128.52.x.x:3260,4 iqn.2010-10.org.openstack:volume-001a 1 | | 67ba5863-9f92-4694-b639-6c9520e0c6f3 | NULL | 128.52.x.x:3260,2 iqn.2010-10.org.openstack:volume-0016 1 | | 7397daa1-f4a7-47d4-b0dc-0b306defdf62 | NULL | 128.52.x.x:3260,14 iqn.2010-10.org.openstack:volume-0014 1 | | 7d8c51bc-9cac-4edf-b1e6-1c37d5a8256f | NULL | 128.52.x.x:3260,10 iqn.2010-10.org.openstack:volume-7d8c51bc-9cac-4edf-b1e6-1c37d5a8256f 1 | | 86426e77-e396-489d-9e66-49f0beef46bb | NULL | 128.52.x.x:3260,16 iqn.2010-10.org.openstack:volume-0019 1 | | 98ac28f5-77d8-476b-b3e1-c90a0fd3e880 | NULL | 128.52.x.x:3260,1 iqn.2010-10.org.openstack:volume-0010 1 | | a6e68eae-23a9-483e-bd42-e4b8a7f47dc4 | NULL | 128.52.x.x:3260,24 iqn.2010-10.org.openstack:volume-a6e68eae-23a9-483e-bd42-e4b8a7f47dc4 1 | | a89b9891-571c-43be-bc1b-0c346a161d38 | NULL | 128.52.x.x:3260,9 iqn.2010-10.org.openstack:volume-a89b9891-571c-43be-bc1b-0c346a161d38 1 | | cbd32221-7794-41d1-abf2-623c49f4ff03 | NULL | 128.52.x.x:3260,6 iqn.2010-10.org.openstack:volume-001b 1 | +--++---+ so I'm also open to suggestions on the right resolution to this. Should I rename the logical volume sand update the provider_location or should I make the /var/lib/cinder/volumes/* files match what is in the database and LVM (and if I do the latter will something come along and undo that)? -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Using nova-volumes openstack LVM group for other pourposes
On Wed, Oct 24, 2012 at 08:56:26PM +0200, Daniel Vázquez wrote: :Hi here! : :Can we create and use news logical volumes for own/custom use(out of :openstack) on nova-volumes openstack LVM group, and use it beside :openstack operational? :IMO it's LVM and no problem, but it has openstack collateral consequences? If you are talking about creating random logical volumes for non-openstack use in the same volume group nova-volume or cinder is using to create volumes (lv are in the same vg but don't otherwise interact), yes you can do that without ocnfusing openstack or having your volumes trampled. For example only having one volume group and using that for operating system partitions as well as volume-foo volumes for cinder I don't think it's a particularly good idea from an organizational standpoint I'd rather have distinct vg's for each purpose so it is clear which resources are operating system and which are data, but in my environment (a private computing/research cloud with a small admin group and 1k users in a few 10's of closely related tenents) it's probably more an aesthetic than technical choice. The larger and more diverse your situation the stronger I'd argue for keeping them in seperate VGs. -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Possible upgrade bug in nova-volume ( cinder)?
On Wed, Oct 24, 2012 at 3:01 PM, John Griffith john.griff...@solidfire.com wrote: Hey Jon, Couple of things going on, one is the volume naming (in progress here: https://review.openstack.org/#/c/14615/). I'll take a closer look at some of the other issues you pointed out. Hi John, On this issue I think the issue you link to above covers my problem (even if that exact implementation now seems abandoned). So the right thing is to normalize all the naming on UUID in my case using lvrename (or symlinks as the patchset above does) and updating the cinder db provider_location. What other issues did you see to look into? Seems this covers everything I brought up here and the issues you helped me with on IRC earlier (thanks) seem like documentation issues (which I hope to get into the docs once I clean up my notes) -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] nova-volume - cinder troubles
Hi All, I'm trying to get from nova-volume to cinder and seem to be tripping up near the end. I have ServerA running as the cloud controller (horizon, keystone, glance, rabbitmq, mysql, nova-api, etc...), ServerB was running nova-volume and is now running cinder. I got the DB created (on ServerA) and pupulated with cinder-manage migrate bits in the release notes. I disabled osapi_volume on ServerB, restarted nova-api and then cinder-api as directed. But api request still seem to be trying (and failing) to use the nova-volume service, as seen by updates to the nova.volumes table when trying to create or attach volumes through Horizon or booting from volume with the python-novaclient. What piece of plubing am I missing? Seem like the controller should be running cinder-api too (but not cinder volume)? -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova-volume - cinder troubles
On Mon, Oct 22, 2012 at 03:06:15PM -0400, Jonathan Proulx wrote: :Hi All, : :I'm trying to get from nova-volume to cinder and seem to be tripping :up near the end. : :I have ServerA running as the cloud controller (horizon, keystone, glance, :rabbitmq, mysql, nova-api, etc...), ServerB was running nova-volume snip :What piece of plubing am I missing? Seem like the controller should be :running cinder-api too (but not cinder volume)? not out of the woods yet, but this problem is that I needed to start the cinder-scheduler process on the controller node ServerA above. -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Folsom nova-scheduler race condition?
On Wed, Oct 10, 2012 at 4:33 AM, Huang Zhiteng winsto...@gmail.com wrote: Yes, but it seems Jonathan's filter list doesn't include RetryFilter, so it's possible that he ran into a race condition that RetryFilter targeted to solve. Yes, that was it exactly. Thank you for seeing the obvious think I was missing... -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Folsom nova-scheduler race condition?
Hi All, Looking for a sanity test before I file a bug. I very recently upgraded my install to Folsom (on top of Ubuntu 12.04/kvm). My scheduler settings in nova.conf are: scheduler_available_filters=nova.scheduler.filters.standard_filters scheduler_default_filters=AvailabilityZoneFilter,RamFilter,CoreFilter,ComputeFilter least_cost_functions=nova.scheduler.least_cost.compute_fill_first_cost_fn compute_fill_first_cost_fn_weight=1.0 cpu_allocation_ratio=1.0 This had been working to fill systems based on available RAM and to not exceed 1:1 allocation ration of CPU resources with Essex. With Folsom, if I specify a moderately large number of instances to boot or spin up single instances in a tight shell loop they will all get schedule on the same compute node well in excess of the number of available vCPUs . If I start them one at a time (using --poll in a shell loop so each instance is started before the next launches) then I get the expected allocation behaviour. I see https://bugs.launchpad.net/nova/+bug/1011852 which seems to attempt to address this issue but as I read it that fix is based on retrying failures. Since KVM is capable of over committing both CPU and Memory I don't seem to get retryable failure, just really bad performance. Am I missing something this this fix or perhaps there's a reported bug I didn't find in my search, or is this really a bug no one has reported? Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Using EC2 through OpenStack
On Tue, Oct 02, 2012 at 12:19:45PM +0200, Michaël Van de Borne wrote: :Hi all, : :This might be obvious, but I can't find the answer. Is there a way to :control EC2 instances using OpenStack? OpenStack provides the same facility as EC2 but on your own hardware, so they don't really touch. Glance (the image store componet of OpenStack) can use S3 for storage, but I don't think it's a good idea unless you have a a really fast pipe to Amazon and deep pockets to pay transfer fees. Perhaps others have different opinions on that point (caveat: I haven't actually tried impeneting this that way) You can use some of the same tools to manage instances on both since OpenStack provides an EC2 compatible API (I've used euca2ools and hybridfox with both). -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] issues with fixed_range blocking new instance creation?
On Thu, Sep 6, 2012 at 9:59 PM, Vishvananda Ishaya vishvana...@gmail.com wrote: fixed range is likely fine. I suspect you created your network with a 10.0.0.0/24 though. It might be a bit tricky to switch to a larger range now. You may have to create the rest of your fixed ips manually in the db associated with the same network and then change the cidr entry on the network in the db to 10.0.0.0/16. You also might have to manually destroy the bridges and let nova-network recreate them. That does seem to be the case, if you'd asked me 24hr ago I'd have sworn I never put /24 anywhere but that's the only explaination that fits observed reality. Luckily I haven't quite called the system production yet so I'm taking a maintenance window now, terminating all instances and deleting and recreating network Being able to expand networks more easily would be a nice thing (and in a dhcp controlled world it's not too disruptive as long as you don't go pushing the gateway address around), but best I can tell this is what I've got right now. Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] After expanding fixed ip range instances no longer have network
Hi All, Running Essex on Ununtu 12.04 using multi-host FlatDHCP nova-networking I ran out of IPs on my fixed_ip range so I shut everything (instances, nova-network, nova-compute) down deleted the old network and recreated it with a smaller netmask. This seems to have almost worked. I can start more instances than I previously had fixed ip's, the right ones seem to be being assigned and the mask on the recreated bridge interfaces is correct, but the (ubuntu-cloudimage) instances can't seem to see their NIC's any more, or perhaps aren't getting dhcp properly I'm still trying to force my way in as our instances rather rely on net access for accounts. On the compute node things seem OK the bridge is up and the right things are connected: root@nova-5:/var/log/nova# brctl show br100 bridge name bridge id STP enabled interfaces br100 8000.60eb69d22521 no eth1 vnet0 vnet1 vnet2 I do notice this in iptables: Chain nova-network-POSTROUTING (1 references) target prot opt source destination ACCEPT all -- 10.0.0.0/16 nova-5.csail.mit.edu ACCEPT all -- 10.0.0.0/16 10.128.0.0/24 ACCEPT all -- 10.0.0.0/16 10.0.0.0/16 ! ctstate DNAT my fixed range is 10.0.0.0/16 not sure where 10.128.0.0/24 comes into it as I don't use that network, but can't see that as a problem. Can any one tell me what I've looked? -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] After expanding fixed ip range instances no longer have network
On Fri, Sep 7, 2012 at 11:33 AM, Jonathan Proulx j...@jonproulx.com wrote: Can any one tell me what I've looked? I assumed stopping and restarting nova-network would restart dnsmasq since dnsmasq doesn't have it's own init script, but this seems not to be the case. dnsmasq is listening on an IP the system nolonger has, I'm sure I'll find the answer ot this on my own soon enough but how does one properly restart openstack's dnsmasq (on ubuntu)? It clearly gets started with lots of command line args from nova.conf -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] After expanding fixed ip range instances no longer have network
On Fri, Sep 07, 2012 at 11:59:20AM -0400, Jonathan Proulx wrote: :On Fri, Sep 7, 2012 at 11:33 AM, Jonathan Proulx j...@jonproulx.com wrote: :dnsmasq is listening on an IP the system nolonger has, I'm sure I'll :find the answer ot this on my own soon enough but how does one :properly restart openstack's dnsmasq (on ubuntu)? It clearly gets :started with lots of command line args from nova.conf For teh record: One way that works is to kill all dnsmasq processes then restart nova-network. Hopefully there's a cleaner way, but I no longer *need* to find it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] After expanding fixed ip range instances no longer have network
Thanks Vish, Your directions were correct would have saved me a few hours of floundering had I just started a little later :) particularly the `killall dnsmasq` -Jon On Fri, Sep 07, 2012 at 09:12:52AM -0700, Vishvananda Ishaya wrote: :If you are sure this is the issue: : :killall dnsmasq :restart nova-network : :Vish : :On Sep 7, 2012, at 8:59 AM, Jonathan Proulx j...@jonproulx.com wrote: : : On Fri, Sep 7, 2012 at 11:33 AM, Jonathan Proulx j...@jonproulx.com wrote: : : Can any one tell me what I've looked? : : I assumed stopping and restarting nova-network would restart dnsmasq : since dnsmasq doesn't have it's own init script, but this seems not to : be the case. : : dnsmasq is listening on an IP the system nolonger has, I'm sure I'll : find the answer ot this on my own soon enough but how does one : properly restart openstack's dnsmasq (on ubuntu)? It clearly gets : started with lots of command line args from nova.conf : : -Jon : : ___ : Mailing list: https://launchpad.net/~openstack : Post to : openstack@lists.launchpad.net : Unsubscribe : https://launchpad.net/~openstack : More help : https://help.launchpad.net/ListHelp : ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] issues with fixed_range blocking new instance creation?
Hi All, my stack stopped launching instances last night I've been chasing my tail for a while but I think it think's it's out of fixed IPs. the system is Essex as packages on Ubuntu 12.04 using FlatDHCP, with multihost and a fixed range or 10.0.0.0/16 --multi_host=True --public_interface=eth0 --fixed_range=10.0.0.0/16 --flat_interface=eth1 --flat_network_bridge=br100 --network_manager=nova.network.manager.FlatDHCPManager the error I'm seeing for all attempts at launching is: 2012-09-06 10:16:43 TRACE nova.compute.manager [instance: d5aa2d4b-76db-43b7-a655-24edeb7fc495] File /usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py, line 69, in _get_configurations 2012-09-06 10:16:43 TRACE nova.compute.manager [instance: d5aa2d4b-76db-43b7-a655-24edeb7fc495] 'ip_address': mapping['ips'][0]['ip'], 2012-09-06 10:16:43 TRACE nova.compute.manager [instance: d5aa2d4b-76db-43b7-a655-24edeb7fc495] IndexError: list index out of range and I just noticed that the flat network interfaces are getting /24 netmasks NOT /16 this combined with 210 running instances and forty some compute nodes make me think it's out of addresses. is my syntax for the fixed range wrong, doc give 10.0.0.0/8 as the example and default so I don't think so. I'm fine with reverting to that, don't mind more addresses, but I would like to understand if this is actually my problem and even if it's not why it's not giving the mask I asked for. -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Instance stuck in deleting state with error
On Wed, Aug 1, 2012 at 3:12 PM, Lorin Hochstein lo...@nimbisservices.com wrote: I believe pip gets it from PyPI: http://pypi.python.org/pypi/python-novaclient/ Ah, I documented this internally and promptly forgot, this is where my version of python-novaclient with reset-state came from: sudo pip install -e git+https://github.com/openstack/python-novaclient.git#egg=python-novaclient you may want to verify the version at http://pypi.python.org/pypi/python-novaclient has it as well. -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Instance stuck in deleting state with error
FYI the nova reset-state command exists on my Debian 6.0 workstations where python-novaclient is installed via pip (presumably the gets latest version from trunk?) it is not available on my Ubuntu 12.04 systems using standard Ubuntu repository versions (essex). -Jon On Wed, Aug 1, 2012 at 2:22 AM, Wolfgang Hennerbichler wolfgang.hennerbich...@risc-software.at wrote: On 07/31/2012 04:17 PM, Rohit Karajgi wrote: Hi Wolfgang, Have you updated the your python-novaclient? no I didn't. The 'nova reset-state server --active' command is pretty much there. It is an admin action in Nova extensions. $ nova help | grep reset reset-state Reset the state of an instance thanks for the hint. Wolfgang Regards, Rohit Karajgi | Lead Engineer | NTT Data Global Technology Services Private Ltd | w. +91.20.6604.1500 x 378 | m. +91 992.242.9639 | rohit.kara...@nttdata.com -Original Message- From: openstack-bounces+rohit.karajgi=nttdata@lists.launchpad.net [mailto:openstack-bounces+rohit.karajgi=nttdata@lists.launchpad.net] On Behalf Of Wolfgang Hennerbichler Sent: Tuesday, July 31, 2012 10:45 AM To: openstack@lists.launchpad.net Subject: Re: [Openstack] Instance stuck in deleting state with error On 07/30/2012 09:35 PM, Kevin L. Mitchell wrote: That said, be aware that there is a reset-state command to novaclient, so that you can do Chris's recommended reset without having to muck around with the database directly. where? nova help | grep reset yields nothing. I think this is one of openstack worst weaknesses, that if the status of an instance is in error-state and one has to wade through a couple of logfiles (scheduler, nova-network, nova-compute) in order to find out what really happened. I would be superior if the error itself would be reported back to the database. Wolfgang ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp __ Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding -- DI (FH) Wolfgang Hennerbichler Software Development Unit Advanced Computing Technologies RISC Software GmbH A company of the Johannes Kepler University Linz IT-Center Softwarepark 35 4232 Hagenberg Austria Phone: +43 7236 3343 245 Fax: +43 7236 3343 250 wolfgang.hennerbich...@risc-software.at http://www.risc-software.at ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Instance stuck in deleting state with error
On Wed, Aug 1, 2012 at 3:40 PM, Lorin Hochstein lo...@nimbisservices.com wrote: What's the use case for resetting an instance to the error state? Is the idea to do: nova reset-state instance nova delete instance That was my use case, though I ended up doing both in the database because I was impatient... ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Instance stuck in deleting state with error
Hi All,
I have an instance that has been in this state for a couple days:
| OS-EXT-STS:power_state | 0|
| OS-EXT-STS:task_state | deleting|
| OS-EXT-STS:vm_state | error |
with this fault:
| fault | {u'message': u'ProcessExecutionError', u'code': 500,
u'created': u'2012-07-30T16:51:34Z'} |
sending more delete requests seem to have no effect and the node it
thinks it is on doesn't seem to have any state about it.
How to I clean this up, do I just need to muck out the DB and if so
what parts of it?
Thanks,
-Jon
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Instance stuck in deleting state with error
Well for posterity or so people can tell me what I did wrong here's
how I seemed to convince the database that the instance was really
deleted:
UPDATE instances SET
vm_state='deleted',task_state='deleted',deleted=1,deleted_at=now()
WHERE uuid=FOO
And here's what I had to do to the volume:
UPDATE volumes SET
attach_status='detached',instance_id=NULL,mountpoint=NULL,status='available'
WHERE id=FOO
Any insight into how I might determine what when wrong here (essex on
ubuntu 12.04 rabbitmq mysql) or if I missed any pieces that will bite
me are appreciated.
-Jon
On Mon, Jul 30, 2012 at 1:03 PM, Jonathan Proulx j...@jonproulx.com wrote:
Hi All,
I have an instance that has been in this state for a couple days:
| OS-EXT-STS:power_state | 0|
| OS-EXT-STS:task_state | deleting|
| OS-EXT-STS:vm_state | error
|
with this fault:
| fault | {u'message': u'ProcessExecutionError', u'code': 500,
u'created': u'2012-07-30T16:51:34Z'} |
sending more delete requests seem to have no effect and the node it
thinks it is on doesn't seem to have any state about it.
How to I clean this up, do I just need to muck out the DB and if so
what parts of it?
Thanks,
-Jon
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Instance stuck in deleting state with error
Thanks but I'm using KVM so not that bug...figures that's the part of my stack I left out. -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Ceph performance as volume image store?
Hi All, I've been looking at Ceph as a storage back end. I'm running a research cluster and while people need to use it and want it 24x7 I don't need as many nines as a commercial customer facing service does so I think I'm OK with the current maturity level as far as that goes, but I have less of a sense of how far along performance is. My OpenStack deployment is 768 cores across 64 physical hosts which I'd like to double in the next 12 months. What it's used for is widely varying and hard to classify some uses are hundreds of tiny nodes others are looking to monopolize the biggest physical system they can get. I think most really heavy IO currently goes to our NAS servers rather than through nova-volumes but that could change. Anyone using ceph at that scale (or preferably larger)? Does it keep up if you keep throwing hardware at it? My proof of concept ceph cluster on crappy salvaged hardware has proved the concept to me but has (unsurprisingly) crappy salvaged performance. Trying to get a sense of what performance expectations I should have given decent hardware before I decide if I should buy decent hardware for it... Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [nova] Specifying Schedule Filters Per Request?
Hi All, http://docs.openstack.org/essex/openstack-compute/admin/content/compute-options-reference.html states that the scheduler_default_filters option in nova.conf controls which filter class names to use for filtering hosts when not specified in the request. How can I specify what filters to use in the request (or can I)? I've been experimenting with DifferentHostFilter and SameHostFilter both of which seem very expensive if left in the default list (even if not passing hints to engage either). It would be nice if these we only enabled for the few cases that want them but I don't see any options to the python-novaclient to pass in a filter list. Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Networking issues in Essex
I've only deployed openstack for the first time a couple weeks ago, but FWIW... I had similar symptoms on my Essex test deployment (on Ubuntu 12.04) turned out my problem was taht while the br100 bridge was up and configured the underlying eth1 physical interface was down so the bits went nowhere. 'ifconfig eth1 up' fixed all, followed ofcoures by fixing in /etc/network/interfaces as well so this happens on it's own in future. -Jon On Thu, Jul 12, 2012 at 02:56:57PM +1000, Michael Chapman wrote: :Hi all, I'm hoping I could get some assistance figuring out my networking :problems with a small Essex test cluster. I have a small Diablo cluster :running without any problems but have hit a wall in deploying Essex. : :I can launch VMs without issue and access them from the compute host, but :from there I can't access anything except the host, DNS services, and other :VMs. : :I have separate machines running keystone, glance, postgresql, rabbit-mq :and nova-api. They're all on the .os domain with 172.22.1.X IPs : :I have one machine running nova-compute, nova-network and nova-api, with a :public address 192.43.239.175 and also an IP on the 172.22.1.X subnet in :the .os domain. It has the following nova/conf: : :--dhcpbridge_flagfile=/etc/nova/nova.conf :--dhcpbridge=/usr/bin/nova-dhcpbridge :--logdir=/var/log/nova :--state_path=/var/lib/nova :--lock_path=/var/lock/nova :--force_dhcp_release :--iscsi_helper=tgtadm :--libvirt_use_virtio_for_bridges :--connection_type=libvirt :--root_helper=sudo nova-rootwrap :--verbose :--ec2_private_dns_show_ip : :--network_manager=nova.network.manager.FlatDHCPManager :--rabbit_host=os-amqp.os :--sql_connection=postgresql://[user]:[password]@os-sql.os/nova :--image_service=nova.image.glance.GlanceImageService :--glance_api_servers=os-glance.os:9292 :--auth_strategy=keystone :--scheduler_driver=nova.scheduler.simple.SimpleScheduler :--keystone_ec2_url=http://os-key.os:5000/v2.0/ec2tokens : :--api_paste_config=/etc/nova/api-paste.ini : :--my_ip=192.43.239.175 :--flat_interface=eth0 :--public_interface=eth1 :--multi_host=True :--routing_source_ip=192.43.239.175 :--network_host=192.43.239.175 : :--dmz_cidr=$my_ip : :--ec2_host=192.43.239.175 :--ec2_dmz_host=192.43.239.175 : :I believe I'm seeing a natting issue of some sort - my VMs cannot ping :external IPs, though DNS seems to work. :ubuntu@monday:~$ ping www.google.com :PING www.l.google.com (74.125.237.148) 56(84) bytes of data. :AWKWARD SILENCE : :When I do a tcpdump on the compute host things seem fairly normal, even :though nothing is getting back to the VM : :root@ncios1:~# tcpdump icmp -i br100 :tcpdump: verbose output suppressed, use -v or -vv for full protocol decode :listening on br100, link-type EN10MB (Ethernet), capture size 65535 bytes :14:35:28.046416 IP 10.0.0.8 syd01s13-in-f20.1e100.net: ICMP echo request, :id 5002, seq 9, length 64 :14:35:28.051477 IP syd01s13-in-f20.1e100.net 10.0.0.8: ICMP echo reply, :id 5002, seq 9, length 64 :14:35:29.054505 IP 10.0.0.8 syd01s13-in-f20.1e100.net: ICMP echo request, :id 5002, seq 10, length 64 :14:35:29.059556 IP syd01s13-in-f20.1e100.net 10.0.0.8: ICMP echo reply, :id 5002, seq 10, length 64 : :I've pored over the iptables nat rules and can't see anything amiss apart :from the masquerades that are automatically added: (I've cut out some empty :chains for brevity) : :root@ncios1:~# iptables -L -t nat -v :Chain PREROUTING (policy ACCEPT 22 packets, 2153 bytes) : pkts bytes target prot opt in out source :destination : 22 2153 nova-network-PREROUTING all -- anyany anywhere : anywhere : 22 2153 nova-compute-PREROUTING all -- anyany anywhere : anywhere : 22 2153 nova-api-PREROUTING all -- anyany anywhere : anywhere : :Chain INPUT (policy ACCEPT 12 packets, 1573 bytes) : pkts bytes target prot opt in out source :destination : :Chain OUTPUT (policy ACCEPT 31 packets, 2021 bytes) : pkts bytes target prot opt in out source :destination : 31 2021 nova-network-OUTPUT all -- anyany anywhere : anywhere : 31 2021 nova-compute-OUTPUT all -- anyany anywhere : anywhere : 31 2021 nova-api-OUTPUT all -- anyany anywhere :anywhere : :Chain POSTROUTING (policy ACCEPT 30 packets, 1961 bytes) : pkts bytes target prot opt in out source :destination : 31 2021 nova-network-POSTROUTING all -- anyany anywhere :anywhere : 30 1961 nova-compute-POSTROUTING all -- anyany anywhere :anywhere : 30 1961 nova-api-POSTROUTING all -- anyany anywhere :anywhere : 30 1961 nova-postrouting-bottom all -- anyany anywhere : anywhere :0 0 MASQUERADE tcp -- anyany 192.168.122.0/24! :192.168.122.0/24 masq ports: 1024-65535 :0 0 MASQUERADE udp -- anyany 192.168.122.0/24! :192.168.122.0/24 masq ports: 1024-65535 :0 0 MASQUERADE all --
Re: [Openstack] [Nova] resource free -vs- allocated utilization?
for posterity yes the info isn't hard to find in the database: mysql select id,vcpus,vcpus_used,memory_mb,memory_mb_used from compute_nodes; I'm not terribly keen on SQL as an interface, guess if it bothers me enough I'll implement a different interface... On Wed, Jul 11, 2012 at 10:34 PM, Jonathan Proulx j...@jonproulx.com wrote: On Wed, Jul 11, 2012 at 8:58 PM, Chris Behrens cbehr...@codestud.com wrote: Hi Jon, There's actually a review up right now proposing to add an OS API extension to be able to give some of this data: https://review.openstack.org/#/c/9544/ that seems to be based on quota limits where as I'm look for just now is over all physical limits for all users. I don't know how you may be looking to query it, but it's not too difficult to get it directly from the instances table in the database, either. I guess I'm looking for something like: eucalyptus-describe-availibility-zones verbose AVAILABILITYZONE |- vm types free / max cpu ram disk AVAILABILITYZONE |- m1.small / 1 128 2 AVAILABILITYZONE |- c1.medium / 1 256 5 AVAILABILITYZONE |- m1.large / 2 512 10 AVAILABILITYZONE |- m1.xlarge / 2 1024 20 AVAILABILITYZONE |- c1.xlarge / 4 2048 20 (well not with the zeros, but it's the first example I could find) where with euca2ools and openstack I get essentially the output of 'nova-manage service list' which is useful but not for the same things. Guess I'll dig into the database shouldn't be too hard to get close to what I want, the eucalyptus output also takes into account fragmentation, which is nice since if I have 100 free cpu slots but on 100 different compute nodes I'm in a bit more trouble than if they are only spread across 10 since in the later case multicore instances can start (for a while anyway). Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [Nova] resource free -vs- allocated utilization?
is there an easy way to see free -vs- allocated nova resources? for example that 300 of 1000 cpu cores are in use (and similar with memory and storage). Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Managing projects roles and quotas (Dashboard -vs- nova-manage)
Hi All, I have a new Essex install on Ubuntu 12.04 and just starting to kick the tires. I would like to manage resource quotas for different projects/tenants but having created them in Dashboard nova-manage doesn't see them, though keystone does and I can join users to tenants with roles using keystone user-role-add. The documentation here http://docs.openstack.org/developer/nova/runnova/managing.projects.html#setting-quotas says to manage projects roles and quotas using 'nova-manage'. Should I avoid the webUI and keystone and use nova-manage for this or is there a different way to manage quotas I'm missing? Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Noob question about api client access
Hi All, I have a shiny new install of openstack and from the controller node (the one with keystone, glance, horizon and all the nova pieces) I can run everything as I expect using the nova cli tools upload images boot instances manage the network, etc. I can also use the Horizon wed interface from remote systems just fine (of course all the actual commands are run locally on the controller) Where it get weird is when I try to use the nova cli on a remote client. Using the same openrc to set up the environment nova credentials returns the expected result but all the other commands I've tried (mostly of the list variety) fail like this: $ nova list ERROR: n/a (HTTP 400) The install is Essex on Ubuntu Precise which I deployed using the Puppet OpenStack module. The example config puppet deployed for me was only listening on 127.0.0.1 so I had to change the bind_host in /etc/keystone/keystone.conf from 127.0.0.1 to 0.0.0.0 to get the 'nova credentials' to run. I've been sifting through the OpenStack Compute Starter Guide and greping for and replacing references to localhost or 127.0.0.1 in the various configs I alsoe tried adding --allow_admin_api=true to the nova.conf which was in the openstack docs but not in my conf, but I'm clearly still missing something. Hopefully this is obvious to some one more familiar with all the pieces? Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
