Re: [one-users] VMs stuck in PEND state
Hi Fernando, you are probably missing the KERNEL and INITRD attributes in your VM template. Take a look at this: http://opennebula.org/documentation:rel2.0:template#os_and_boot_options_section Regards, Jaime On Wed, Nov 10, 2010 at 5:39 PM, Fernando Morgenstern ferna...@consultorpc.com wrote: Hello, After applying this patch and adding host again i had a couple of different errors ( like ruby not being installed in the node ) which i was able to fix. But now i got stuck again during the VM startup, at the log i see the following error: Wed Nov 10 17:22:38 2010 [LCM][I]: New VM state is BOOT Wed Nov 10 17:22:38 2010 [VMM][I]: Generating deployment file: /var/lib/one/5/deployment.0 Wed Nov 10 17:22:38 2010 [VMM][E]: No kernel or bootloader defined and no default provided. Wed Nov 10 17:22:38 2010 [VMM][E]: deploy_action, error generating deployment file: /var/lib/one/5/deployment.0 Wed Nov 10 17:22:38 2010 [DiM][I]: New VM state is FAILED Wed Nov 10 17:22:38 2010 [TM][W]: Ignored: LOG - 5 tm_delete.sh: Deleting /var/lib/one//5/images Wed Nov 10 17:22:38 2010 [TM][W]: Ignored: LOG - 5 tm_delete.sh: Executed rm -rf /var/lib/one//5/images. Wed Nov 10 17:22:38 2010 [TM][W]: Ignored: TRANSFER SUCCESS 5 - The thread that i found here at list about this doesn't contain a solution for this problem, so i'm not sure how to proceed. Regards, Fernando. Em 10/11/2010, às 11:49, openneb...@nerling.ch escreveu: Yes, it is related to the bug http://dev.opennebula.org/issues/385. I attached a patch for /usr/lib/one/mads/one_im_ssh.rb. ## Save it on the /tmp #cd /usr/lib/one/mads/ #patch -p0 /tmp/one_im_ssh.rb.patch Quoting Fernando Morgenstern ferna...@consultorpc.com: Hi, The tmp folder in my host is empty. Here is the output of the commands: $ onehost show 1 HOST 1 INFORMATION ID: 1 NAME : node01 CLUSTER : default STATE : ERROR IM_MAD: im_xen VM_MAD: vmm_xen TM_MAD: tm_nfs HOST SHARES MAX MEM : 0 USED MEM (REAL) : 0 USED MEM (ALLOCATED) : 0 MAX CPU : 0 USED CPU (REAL) : 0 USED CPU (ALLOCATED) : 0 RUNNING VMS : 0 MONITORING INFORMATION $ onehost show -x 1 HOST ID1/ID NAMEnode01/NAME STATE3/STATE IM_MADim_xen/IM_MAD VM_MADvmm_xen/VM_MAD TM_MADtm_nfs/TM_MAD LAST_MON_TIME1289394482/LAST_MON_TIME CLUSTERdefault/CLUSTER HOST_SHARE HID1/HID DISK_USAGE0/DISK_USAGE MEM_USAGE0/MEM_USAGE CPU_USAGE0/CPU_USAGE MAX_DISK0/MAX_DISK MAX_MEM0/MAX_MEM MAX_CPU0/MAX_CPU FREE_DISK0/FREE_DISK FREE_MEM0/FREE_MEM FREE_CPU0/FREE_CPU USED_DISK0/USED_DISK USED_MEM0/USED_MEM USED_CPU0/USED_CPU RUNNING_VMS0/RUNNING_VMS /HOST_SHARE TEMPLATE/ /HOST Thanks again for your answers. Em 10/11/2010, às 11:22, openneb...@nerling.ch escreveu: Hallo Fernando. try to log in the host and look if there is a folder names /tmp/one. If not it could be related to the bug: http://dev.opennebula.org/issues/385 please post the output from: #onehost show 1 #onehost show -x 1 I thought before your host has an id of 0. Marlon Nerling Quoting Fernando Morgenstern ferna...@consultorpc.com: Hello, Thanks for the answer. You are right, the host is showing an error state and i didn't verified it. How can i know what is causing the error in host? $ onehost list ID NAME CLUSTER RVM TCPU FCPU ACPUTMEMFMEM STAT 1 node01default0 0 0100 0K 0K err $ onevm show 0 VIRTUAL MACHINE 0 INFORMATION ID : 0 NAME : ttylinux STATE : DONE LCM_STATE : LCM_INIT START TIME : 11/09 19:06:37 END TIME : 11/09 19:11:09 DEPLOY ID: : - VIRTUAL MACHINE MONITORING NET_RX : 0 USED MEMORY: 0 USED CPU : 0 NET_TX : 0 VIRTUAL MACHINE TEMPLATE CPU=0.1 DISK=[ DISK_ID=0, READONLY=no, SOURCE=/home/oneadmin/ttylinux.img, TARGET=hda ] FEATURES=[ ACPI=no ] MEMORY=64 NAME=ttylinux NIC=[ BRIDGE=br0, IP=*, MAC=02:00:5d:9f:d0:68, NETWORK=Small network, NETWORK_ID=0 ] VMID=0 $ onehost show 0 Error: [HostInfo] Error getting HOST [0]. Thanks! Em 10/11/2010, às 06:45, openneb...@nerling.ch escreveu: Hallo Fernando. Could you please post the output of: #onehost list #onevm show 0 #onehost show 0 It seems that none of your Hosts are enabled! Tue Nov 9 20:31:18 2010 [HOST][D]: Discovered Hosts (enabled): best regards Marlon Nerling Zitat von Fernando Morgenstern ferna...@consultorpc.com: Hello, This is the first time that i'm using open nebula, so i tried to do it with express script which ran fine. I'm using CentOS 5.5 with
Re: [one-users] Making scheduler allocation aware
Hallo Shashank.
I'm having the same problem in 1.4.
You must workaround it by youself, so instead of using onevm directly,
use it with a wrapper script who looks and waits for deploy if a VM is
pending.
I hope this behaviour is fixed on 2.0 (Hallo developers??)
Best regards
Marlon
Zitat von Shashank Rachamalla shashank.rachama...@hexagrid.com:
Hi Javier
Thanks for the inputs but I came across another problem while testing:
If opennebula receives multiple vm requests in a short span of time, the
scheduler might take decisions for all these vms considering the host
monitoring information available from the last monitoring cycle. Ideally,
before processing every pending request, fresh host monitoring information
has to be taken into account as the previous set of requests might have
already changed the host’s state. This can result in over committing when
host is being used close to its full capacity.
*Is there any workaround which helps the scheduler to overcome the above
problem ?*
steps to reproduce the problem scenario:
Host 1 : Total memory = 3GB
Host 2 : Total memory = 2GB
Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have a
higher RANK value )
VM1: memory = 2GB
VM2: memroy = 2GB
Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will
come up on Host1. ( Thus over committing )
Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up on
Host1 and VM2 will come up on Host2. This is true because opennebula would
have fetched a fresh set of host monitoring information in that time.
On 4 November 2010 02:04, Javier Fontan jfon...@gmail.com wrote:
Hello,
It looks fine to me. I think that taking out the memory the hypervisor
may be consuming is key to make it work.
Bye
On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
rb...@hexagrid.com wrote:
Javier
Yes we are using KVM and OpenNebula 1.4.
We have been having this problem since a long time and we were doing all
kinds of validations ourselves before submitting the request to
OpenNebula.
(there should be enough memory in the cloud that matches the requested
memory there should be atleast one host that has memory requested
memory
) We had to do those because OpenNebula would schedule to an arbitrary
host based on the existing logic it had.
So at last we thought that we need to make OpenNebula aware of memory
allocated of running VM's on the host and started this discussion.
Thanks for taking up this issue as priority. Appreciate it.
Shashank came up with this patch to kvm.rb. Please take a look and let us
know if that will work until we get a permanent solution.
$mem_allocated_for_running_vms=0
for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
$dominfo=`virsh dominfo #{i}`
$dominfo.split(/\n/).each{|line|
if line.match('^Max memory')
$mem_allocated_for_running_vms += line.split(
)[2].strip.to_i
end
}
end
$mem_used_by_base_hypervisor = [some xyz kb that we want to set aside for
hypervisor]
$free_memory = $total_memory.to_i - ( $mem_allocated_for_running_vms.to_i
+
$mem_used_by_base_hypervisor.to_i )
==
Ranga
On Wed, Nov 3, 2010 at 2:16 PM, Javier Fontan jfon...@gmail.com wrote:
Hello,
Sorry for the delay in the response.
It looks that the problem is OpenNebula calculating available memory.
For xen = 3.2 there is a reliable way to get available memory that is
calling xm info and getting max_free_memory attribute.
Unfortunately for kvm or xen 3.2 there is not such attribute. I
suppose you are using kvm as you tell about free command.
I began analyzing the kvm IM probe that gets memory information and
there is a problem on the way to get total memory. Here is how it now
gets memory information:
TOTALMEMORY: runs virsh info that gets the real physical memory
installed in the machine
FREEMEMORY: runs free command and gets the free column data without
buffers and cache
USEDMEMORY: runs top command and gets used memory from it (this counts
buffers and cache)
This is a big problem as those values do not match one with another (I
don't really know how I failed to see this before). Here is the
monitoring data from a host without VMs.
--8--
TOTALMEMORY=8193988
USEDMEMORY=7819952
FREEMEMORY=7911924
--8--
As you can see it makes no sense at all. Even the TOTALMEMORY that is
got from virsh info is very misleading for oned as the host linux
instance does not have access to all that memory (some is consumed by
the hypervisor itself) as seen calling a free command:
--8--
total used free sharedbuffers cached
Mem: 81939887819192 374796 0 64176
7473992
--8--
I am also copying this text as an issue
Re: [one-users] Making scheduler allocation aware
Hi,
Regarding capacity (CPU, Memory), it is updated every time a VM is
submitted, so no overcommitment is possible (beyond that specified by
the CPU attribute in the VM template). This also works in 1.4
Cheers
Ruben
On Thu, Nov 11, 2010 at 10:08 AM, openneb...@nerling.ch wrote:
Hallo Shashank.
I'm having the same problem in 1.4.
You must workaround it by youself, so instead of using onevm directly, use
it with a wrapper script who looks and waits for deploy if a VM is pending.
I hope this behaviour is fixed on 2.0 (Hallo developers??)
Best regards
Marlon
Zitat von Shashank Rachamalla shashank.rachama...@hexagrid.com:
Hi Javier
Thanks for the inputs but I came across another problem while testing:
If opennebula receives multiple vm requests in a short span of time, the
scheduler might take decisions for all these vms considering the host
monitoring information available from the last monitoring cycle. Ideally,
before processing every pending request, fresh host monitoring
information
has to be taken into account as the previous set of requests might have
already changed the host’s state. This can result in over committing when
host is being used close to its full capacity.
*Is there any workaround which helps the scheduler to overcome the above
problem ?*
steps to reproduce the problem scenario:
Host 1 : Total memory = 3GB
Host 2 : Total memory = 2GB
Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have a
higher RANK value )
VM1: memory = 2GB
VM2: memroy = 2GB
Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will
come up on Host1. ( Thus over committing )
Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up on
Host1 and VM2 will come up on Host2. This is true because opennebula would
have fetched a fresh set of host monitoring information in that time.
On 4 November 2010 02:04, Javier Fontan jfon...@gmail.com wrote:
Hello,
It looks fine to me. I think that taking out the memory the hypervisor
may be consuming is key to make it work.
Bye
On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
rb...@hexagrid.com wrote:
Javier
Yes we are using KVM and OpenNebula 1.4.
We have been having this problem since a long time and we were doing
all
kinds of validations ourselves before submitting the request to
OpenNebula.
(there should be enough memory in the cloud that matches the requested
memory there should be atleast one host that has memory requested
memory
) We had to do those because OpenNebula would schedule to an
arbitrary
host based on the existing logic it had.
So at last we thought that we need to make OpenNebula aware of memory
allocated of running VM's on the host and started this discussion.
Thanks for taking up this issue as priority. Appreciate it.
Shashank came up with this patch to kvm.rb. Please take a look and let
us
know if that will work until we get a permanent solution.
$mem_allocated_for_running_vms=0
for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
$dominfo=`virsh dominfo #{i}`
$dominfo.split(/\n/).each{|line|
if line.match('^Max memory')
$mem_allocated_for_running_vms += line.split(
)[2].strip.to_i
end
}
end
$mem_used_by_base_hypervisor = [some xyz kb that we want to set aside
for
hypervisor]
$free_memory = $total_memory.to_i - (
$mem_allocated_for_running_vms.to_i
+
$mem_used_by_base_hypervisor.to_i )
==
Ranga
On Wed, Nov 3, 2010 at 2:16 PM, Javier Fontan jfon...@gmail.com
wrote:
Hello,
Sorry for the delay in the response.
It looks that the problem is OpenNebula calculating available memory.
For xen = 3.2 there is a reliable way to get available memory that is
calling xm info and getting max_free_memory attribute.
Unfortunately for kvm or xen 3.2 there is not such attribute. I
suppose you are using kvm as you tell about free command.
I began analyzing the kvm IM probe that gets memory information and
there is a problem on the way to get total memory. Here is how it now
gets memory information:
TOTALMEMORY: runs virsh info that gets the real physical memory
installed in the machine
FREEMEMORY: runs free command and gets the free column data without
buffers and cache
USEDMEMORY: runs top command and gets used memory from it (this counts
buffers and cache)
This is a big problem as those values do not match one with another (I
don't really know how I failed to see this before). Here is the
monitoring data from a host without VMs.
--8--
TOTALMEMORY=8193988
USEDMEMORY=7819952
FREEMEMORY=7911924
--8--
As you can see it makes no sense at all. Even the TOTALMEMORY that
Re: [one-users] Making scheduler allocation aware
Hi,
We use the total memory to allocate VM's, this is not going to change
between monitoring events. Right now, we are able to schedule multiple
VMs in the same scheduling step, to the same host without
overcommitting memory.
Cheers
On Thu, Nov 11, 2010 at 10:42 AM, Shashank Rachamalla
shashank.rachama...@hexagrid.com wrote:
Hi
Thanks for the reply. I agree that a VM will not be allowed to use more than
what has been specified in its template during creation but I was referring
to a scenario where memory available on the host is over committed. I guess
the problem here is with allowing multiple VMs to be dispatched on to a
single host in one schedule cycle.
On 11 November 2010 14:47, Ruben S. Montero rube...@dacya.ucm.es wrote:
Hi,
Regarding capacity (CPU, Memory), it is updated every time a VM is
submitted, so no overcommitment is possible (beyond that specified by
the CPU attribute in the VM template). This also works in 1.4
Cheers
Ruben
On Thu, Nov 11, 2010 at 10:08 AM, openneb...@nerling.ch wrote:
Hallo Shashank.
I'm having the same problem in 1.4.
You must workaround it by youself, so instead of using onevm directly,
use
it with a wrapper script who looks and waits for deploy if a VM is
pending.
I hope this behaviour is fixed on 2.0 (Hallo developers??)
Best regards
Marlon
Zitat von Shashank Rachamalla shashank.rachama...@hexagrid.com:
Hi Javier
Thanks for the inputs but I came across another problem while testing:
If opennebula receives multiple vm requests in a short span of time,
the
scheduler might take decisions for all these vms considering the host
monitoring information available from the last monitoring cycle.
Ideally,
before processing every pending request, fresh host monitoring
information
has to be taken into account as the previous set of requests might have
already changed the host’s state. This can result in over committing
when
host is being used close to its full capacity.
*Is there any workaround which helps the scheduler to overcome the
above
problem ?*
steps to reproduce the problem scenario:
Host 1 : Total memory = 3GB
Host 2 : Total memory = 2GB
Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have
a
higher RANK value )
VM1: memory = 2GB
VM2: memroy = 2GB
Start VM1 and VM2 immediately one after the other. Both VM1 and VM2
will
come up on Host1. ( Thus over committing )
Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up
on
Host1 and VM2 will come up on Host2. This is true because opennebula
would
have fetched a fresh set of host monitoring information in that time.
On 4 November 2010 02:04, Javier Fontan jfon...@gmail.com wrote:
Hello,
It looks fine to me. I think that taking out the memory the hypervisor
may be consuming is key to make it work.
Bye
On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
rb...@hexagrid.com wrote:
Javier
Yes we are using KVM and OpenNebula 1.4.
We have been having this problem since a long time and we were doing
all
kinds of validations ourselves before submitting the request to
OpenNebula.
(there should be enough memory in the cloud that matches the
requested
memory there should be atleast one host that has memory
requested
memory
) We had to do those because OpenNebula would schedule to an
arbitrary
host based on the existing logic it had.
So at last we thought that we need to make OpenNebula aware of
memory
allocated of running VM's on the host and started this discussion.
Thanks for taking up this issue as priority. Appreciate it.
Shashank came up with this patch to kvm.rb. Please take a look and
let
us
know if that will work until we get a permanent solution.
$mem_allocated_for_running_vms=0
for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
$dominfo=`virsh dominfo #{i}`
$dominfo.split(/\n/).each{|line|
if line.match('^Max memory')
$mem_allocated_for_running_vms += line.split(
)[2].strip.to_i
end
}
end
$mem_used_by_base_hypervisor = [some xyz kb that we want to set
aside
for
hypervisor]
$free_memory = $total_memory.to_i - (
$mem_allocated_for_running_vms.to_i
+
$mem_used_by_base_hypervisor.to_i )
==
Ranga
On Wed, Nov 3, 2010 at 2:16 PM, Javier Fontan jfon...@gmail.com
wrote:
Hello,
Sorry for the delay in the response.
It looks that the problem is OpenNebula calculating available
memory.
For xen = 3.2 there is a reliable way to get available memory that
is
calling xm info and getting max_free_memory attribute.
Re: [one-users] Making scheduler allocation aware
Hi Rubens.
I'm sure the RANK=-RUNNING_VMS will not apply in such a scenario,
because the scheduler does not update the RUNNING_VMS after the
creation of the VM but just after monitoring the host!
So between this events the RUNNING_VMS value stays unchanged and by my
experience this Host will become the 'chosen one' for new deployed VMs
up to the next host monitoring.
And i'm not really sure if the scheduler sums USED MEMORY (ALLOCATED)
and the memory used by the VM to prevent overcommiting, we could look
in the source code for this.
I must say never have experienced a host who became more VMs as possible.
Best regards
Marlon
Zitat von Ruben S. Montero rube...@dacya.ucm.es:
Hi,
We use the total memory to allocate VM's, this is not going to change
between monitoring events. Right now, we are able to schedule multiple
VMs in the same scheduling step, to the same host without
overcommitting memory.
Cheers
On Thu, Nov 11, 2010 at 10:42 AM, Shashank Rachamalla
shashank.rachama...@hexagrid.com wrote:
Hi
Thanks for the reply. I agree that a VM will not be allowed to use more than
what has been specified in its template during creation but I was referring
to a scenario where memory available on the host is over committed. I guess
the problem here is with allowing multiple VMs to be dispatched on to a
single host in one schedule cycle.
On 11 November 2010 14:47, Ruben S. Montero rube...@dacya.ucm.es wrote:
Hi,
Regarding capacity (CPU, Memory), it is updated every time a VM is
submitted, so no overcommitment is possible (beyond that specified by
the CPU attribute in the VM template). This also works in 1.4
Cheers
Ruben
On Thu, Nov 11, 2010 at 10:08 AM, openneb...@nerling.ch wrote:
Hallo Shashank.
I'm having the same problem in 1.4.
You must workaround it by youself, so instead of using onevm directly,
use
it with a wrapper script who looks and waits for deploy if a VM is
pending.
I hope this behaviour is fixed on 2.0 (Hallo developers??)
Best regards
Marlon
Zitat von Shashank Rachamalla shashank.rachama...@hexagrid.com:
Hi Javier
Thanks for the inputs but I came across another problem while testing:
If opennebula receives multiple vm requests in a short span of time,
the
scheduler might take decisions for all these vms considering the host
monitoring information available from the last monitoring cycle.
Ideally,
before processing every pending request, fresh host monitoring
information
has to be taken into account as the previous set of requests might have
already changed the host’s state. This can result in over committing
when
host is being used close to its full capacity.
*Is there any workaround which helps the scheduler to overcome the
above
problem ?*
steps to reproduce the problem scenario:
Host 1 : Total memory = 3GB
Host 2 : Total memory = 2GB
Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have
a
higher RANK value )
VM1: memory = 2GB
VM2: memroy = 2GB
Start VM1 and VM2 immediately one after the other. Both VM1 and VM2
will
come up on Host1. ( Thus over committing )
Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up
on
Host1 and VM2 will come up on Host2. This is true because opennebula
would
have fetched a fresh set of host monitoring information in that time.
On 4 November 2010 02:04, Javier Fontan jfon...@gmail.com wrote:
Hello,
It looks fine to me. I think that taking out the memory the hypervisor
may be consuming is key to make it work.
Bye
On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
rb...@hexagrid.com wrote:
Javier
Yes we are using KVM and OpenNebula 1.4.
We have been having this problem since a long time and we were doing
all
kinds of validations ourselves before submitting the request to
OpenNebula.
(there should be enough memory in the cloud that matches the
requested
memory there should be atleast one host that has memory
requested
memory
) We had to do those because OpenNebula would schedule to an
arbitrary
host based on the existing logic it had.
So at last we thought that we need to make OpenNebula aware of
memory
allocated of running VM's on the host and started this discussion.
Thanks for taking up this issue as priority. Appreciate it.
Shashank came up with this patch to kvm.rb. Please take a look and
let
us
know if that will work until we get a permanent solution.
$mem_allocated_for_running_vms=0
for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
$dominfo=`virsh dominfo #{i}`
$dominfo.split(/\n/).each{|line|
if line.match('^Max memory')
$mem_allocated_for_running_vms += line.split(
)[2].strip.to_i
end
}
end
$mem_used_by_base_hypervisor = [some xyz kb that we want to set
aside
for
hypervisor]
Re: [one-users] Making scheduler allocation aware
Hi What Marlon said regarding RANK=RUNNING_VMS is the exact concern. However, I will check out the source code once. We are using opennebula 1.4 with the following template: ( Note that the value of PRIORITY is calculated using FREEMEMORY, FREECPU and RUNNING_VMS ) DISK=[ clone=no, source=/mnt/dev_store_10/images/glpnnu0, target=hda, type=disk ] DISK=[ clone=no, source=/mnt/dev_store_10/iso/FABCAC11-768B-4683-EB99-085ECB80, target=hdb, type=cdrom ] MEMORY=2000 REQUIREMENTS=FREEMEMORY2048000 RANK=PRIORITY NAME=glpnnu OS=[ boot=cdrom ] NIC=[ model=e1000, bridge=br101, MAC=00:89:10:36:10:26 ] INPUT=[ type=tablet ] VCPU=1 GRAPHICS=[ port=5971, type=vnc, listen=0.0.0.0 ] On 11 November 2010 16:08, openneb...@nerling.ch wrote: Hi Rubens. I'm sure the RANK=-RUNNING_VMS will not apply in such a scenario, because the scheduler does not update the RUNNING_VMS after the creation of the VM but just after monitoring the host! So between this events the RUNNING_VMS value stays unchanged and by my experience this Host will become the 'chosen one' for new deployed VMs up to the next host monitoring. And i'm not really sure if the scheduler sums USED MEMORY (ALLOCATED) and the memory used by the VM to prevent overcommiting, we could look in the source code for this. I must say never have experienced a host who became more VMs as possible. Best regards Marlon Zitat von Ruben S. Montero rube...@dacya.ucm.es: Hi, We use the total memory to allocate VM's, this is not going to change between monitoring events. Right now, we are able to schedule multiple VMs in the same scheduling step, to the same host without overcommitting memory. Cheers On Thu, Nov 11, 2010 at 10:42 AM, Shashank Rachamalla shashank.rachama...@hexagrid.com wrote: Hi Thanks for the reply. I agree that a VM will not be allowed to use more than what has been specified in its template during creation but I was referring to a scenario where memory available on the host is over committed. I guess the problem here is with allowing multiple VMs to be dispatched on to a single host in one schedule cycle. On 11 November 2010 14:47, Ruben S. Montero rube...@dacya.ucm.es wrote: Hi, Regarding capacity (CPU, Memory), it is updated every time a VM is submitted, so no overcommitment is possible (beyond that specified by the CPU attribute in the VM template). This also works in 1.4 Cheers Ruben On Thu, Nov 11, 2010 at 10:08 AM, openneb...@nerling.ch wrote: Hallo Shashank. I'm having the same problem in 1.4. You must workaround it by youself, so instead of using onevm directly, use it with a wrapper script who looks and waits for deploy if a VM is pending. I hope this behaviour is fixed on 2.0 (Hallo developers??) Best regards Marlon Zitat von Shashank Rachamalla shashank.rachama...@hexagrid.com: Hi Javier Thanks for the inputs but I came across another problem while testing: If opennebula receives multiple vm requests in a short span of time, the scheduler might take decisions for all these vms considering the host monitoring information available from the last monitoring cycle. Ideally, before processing every pending request, fresh host monitoring information has to be taken into account as the previous set of requests might have already changed the host’s state. This can result in over committing when host is being used close to its full capacity. *Is there any workaround which helps the scheduler to overcome the above problem ?* steps to reproduce the problem scenario: Host 1 : Total memory = 3GB Host 2 : Total memory = 2GB Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have a higher RANK value ) VM1: memory = 2GB VM2: memroy = 2GB Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will come up on Host1. ( Thus over committing ) Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up on Host1 and VM2 will come up on Host2. This is true because opennebula would have fetched a fresh set of host monitoring information in that time. On 4 November 2010 02:04, Javier Fontan jfon...@gmail.com wrote: Hello, It looks fine to me. I think that taking out the memory the hypervisor may be consuming is key to make it work. Bye On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula rb...@hexagrid.com wrote: Javier Yes we are using KVM and OpenNebula 1.4. We have been having this problem since a long time and we were doing all kinds of validations ourselves before submitting the request to OpenNebula. (there should be enough memory in the cloud that matches the requested memory there should be atleast one host that has memory requested memory ) We had to
Re: [one-users] Fail-over from dead HV node
Dear Oscar, We are working on an extension of the hooks mechanism, so when a host gets into the ERROR state a script can be triggered with information about the (allegedly) running VMs in that host, so it can resume them elsewhere. Best regards, -Tino -- Constantino Vázquez Blanco | dsa-research.org/tinova Virtualization Technology Engineer / Researcher OpenNebula Toolkit | opennebula.org On Fri, Oct 29, 2010 at 12:03 PM, Oscar Elfving sofa...@gmail.com wrote: Hi, When a hypervisor node dies (or becomes inaccessible), how are you supposed to handle the machines that were running on it? ONE of course still lists the machines as running since it can't check the machine state. How are you guys handling it? In my case I have some single purpose webservers running on the nodes and if a hypervisor node dies, I would like to have some monitoring software start them up on one of the remaining nodes. Best regards, Oscar Elfving ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Fail-over from dead HV node
On 2010. November 11. 15:25:33 Tino Vazquez wrote: We are working on an extension of the hooks mechanism, so when a host gets into the ERROR state a script can be triggered with information about the (allegedly) running VMs in that host, so it can resume them elsewhere. How do you plan to distinguish VMs that crashed due to host hardware failure from the ones those were shut down by their administrator from inside the WM? Thanks, -- cc -- Constantino Vázquez Blanco | dsa-research.org/tinova Virtualization Technology Engineer / Researcher OpenNebula Toolkit | opennebula.org On Fri, Oct 29, 2010 at 12:03 PM, Oscar Elfving sofa...@gmail.com wrote: Hi, When a hypervisor node dies (or becomes inaccessible), how are you supposed to handle the machines that were running on it? ONE of course still lists the machines as running since it can't check the machine state. How are you guys handling it? In my case I have some single purpose webservers running on the nodes and if a hypervisor node dies, I would like to have some monitoring software start them up on one of the remaining nodes. Best regards, Oscar Elfving ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
[one-users] sparse files as partitions in OpenNebula?
I have a user who has a need for 250GB of disk storage (eventually) that he would like to migrate around with his VM. NFS isn't suitable for this application. This is an application which will start with a file base and then gradually grow. On Amazon this could be a use case for EBS but ONE doesn't have anything like that as far as I can tell. My question, can I create an opennebula template that calls out device vdc as a sparse file system eventually growable to 250 GB, and migrate that and save that as necessary? If so, how? We are running opennebula 2.0 and using KVM as our hypervisor. Steve Timm -- -- Steven C. Timm, Ph.D (630) 840-8525 t...@fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division, Scientific Computing Facilities, Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader. ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
[one-users] [one] opennebula with VMware server 2.0
All, I am trying to Installing opennebula with following combinations Opennebula 2.0 (Ubuntu 10.04) - NFS - VMware server 2.0 I have followed the VMware Driver Guide and installed Libvirt 0.8.3, VMware Server and OpenNebula http://opennebula.org/documentation:rel2.0:evmwareg Now when I am trying to configure the opennebula for VMware server, its not working, I could not able to figure it out where is the issue. So I have couple of question here 1. How to check the complete connectivity between opennebula - libvirt - VMware server for its correctness. 2. Cofiguration documents are not clear e.g in VMware server vmwarerc file should be gsx:// rather esx:// which is never indicated anywhere. So is there any document which talks about all these in finer details. Any kind of help will be appreciated. Thanks -tej ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
[one-users] Examples for using Ruby API
Hi, I am new to opennebula and trying to understand how to use ruby api. I know little scripting. I am looking for some basic examples for opennebula setup info like onehost list, onevm list etc using ruby api. My goal is to build basic web frontend to automate certain cli task using ruby api. Regards ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Examples for using Ruby API
On Thursday 11 November 2010 20:42:00 KONGQUERLABS wrote: Hi, I am new to opennebula and trying to understand how to use ruby api. I know little scripting. I am looking for some basic examples for opennebula setup info like onehost list, onevm list etc using ruby api. All of them are already written in ruby api;) Regards, Łukasz Oleś ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Examples for using Ruby API
I realize that , how can i use the ruby api from remote systems. On Fri, Nov 12, 2010 at 3:02 AM, Łukasz Oleś lukaszo...@gmail.com wrote: On Thursday 11 November 2010 20:42:00 KONGQUERLABS wrote: Hi, I am new to opennebula and trying to understand how to use ruby api. I know little scripting. I am looking for some basic examples for opennebula setup info like onehost list, onevm list etc using ruby api. All of them are already written in ruby api;) Regards, Łukasz Oleś ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Examples for using Ruby API
If you are using OpenNebula 2.0 you can use the oca gem in your ruby project. https://rubygems.org/gems/oca The API is documented to a certain degree, but missing examples. You can always look at the oca source in the git repo (src/oca/ruby/test/). John On Thu, Nov 11, 2010 at 6:31 PM, KONQUER LABS konquerl...@gmail.com wrote: I realize that , how can i use the ruby api from remote systems. On Fri, Nov 12, 2010 at 3:02 AM, Łukasz Oleś lukaszo...@gmail.com wrote: On Thursday 11 November 2010 20:42:00 KONGQUERLABS wrote: Hi, I am new to opennebula and trying to understand how to use ruby api. I know little scripting. I am looking for some basic examples for opennebula setup info like onehost list, onevm list etc using ruby api. All of them are already written in ruby api;) Regards, Łukasz Oleś ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Making scheduler allocation aware
Hi
As Ruben pointed out, every time a VM is dispatched, host share counter (
mem_usage ) gets incremented. Thus there is no way for opennebula to allow
over committing. However, the problem lies in the following piece of code
from VirtualMachine.cc where cpu, memory and disk are set to 0 when a VM
template does not contain CPU attribute ( this is true in our case ). The
mem_usage value in host share does not get incremented because of this. I
guess, specifying CPU=0 in the template should solve the problem. I will
reconfirm after testing.
void VirtualMachine::get_requirements (int cpu, int memory, int disk)
{
string scpu;
istringstream iss;
float fcpu;
get_template_attribute(MEMORY,memory);
get_template_attribute(CPU,scpu);
*if ((memory == 0) || (scpu==))
{
cpu= 0;
memory = 0;
disk = 0;
return;
}
*
if(scpu==)
{
scpu=0;
}
iss.str(scpu);
iss fcpu;
cpu= (int) (fcpu * 100);//now in 100%
memory = memory * 1024; //now in bytes
disk = 0;
return;
}
On 11 November 2010 19:36, Javier Fontan jfon...@gmail.com wrote:
Hello,
Are you sure that those are the exact values for the host? OpenNebula
counts real (from probes) and allocated (from database) memory so
that should not happen.
Snippet from a onehost show:
--8--
USED MEM (REAL) : 0
USED MEM (ALLOCATED) : 65536
--8--
I am now working on the kvm monitoring and I have noticed another
mismatch even with your probe changes. The values stored in the
database for total memory should be changed and that's what I am
working on.
I am connected to irc.freenode.org in channel #opennebula if you want
to discuss this further.
Bye
On Thu, Nov 11, 2010 at 5:20 AM, Shashank Rachamalla
shashank.rachama...@hexagrid.com wrote:
Hi Javier
Thanks for the inputs but I came across another problem while testing:
If opennebula receives multiple vm requests in a short span of time, the
scheduler might take decisions for all these vms considering the host
monitoring information available from the last monitoring cycle. Ideally,
before processing every pending request, fresh host monitoring
information
has to be taken into account as the previous set of requests might have
already changed the host’s state. This can result in over committing when
host is being used close to its full capacity.
Is there any workaround which helps the scheduler to overcome the above
problem ?
steps to reproduce the problem scenario:
Host 1 : Total memory = 3GB
Host 2 : Total memory = 2GB
Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have a
higher RANK value )
VM1: memory = 2GB
VM2: memroy = 2GB
Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will
come up on Host1. ( Thus over committing )
Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up
on
Host1 and VM2 will come up on Host2. This is true because opennebula
would
have fetched a fresh set of host monitoring information in that time.
On 4 November 2010 02:04, Javier Fontan jfon...@gmail.com wrote:
Hello,
It looks fine to me. I think that taking out the memory the hypervisor
may be consuming is key to make it work.
Bye
On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
rb...@hexagrid.com wrote:
Javier
Yes we are using KVM and OpenNebula 1.4.
We have been having this problem since a long time and we were doing
all
kinds of validations ourselves before submitting the request to
OpenNebula.
(there should be enough memory in the cloud that matches the
requested
memory there should be atleast one host that has memory requested
memory
) We had to do those because OpenNebula would schedule to an
arbitrary
host based on the existing logic it had.
So at last we thought that we need to make OpenNebula aware of memory
allocated of running VM's on the host and started this discussion.
Thanks for taking up this issue as priority. Appreciate it.
Shashank came up with this patch to kvm.rb. Please take a look and let
us
know if that will work until we get a permanent solution.
$mem_allocated_for_running_vms=0
for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
$dominfo=`virsh dominfo #{i}`
$dominfo.split(/\n/).each{|line|
if line.match('^Max memory')
$mem_allocated_for_running_vms += line.split(
)[2].strip.to_i
end
}
end
$mem_used_by_base_hypervisor = [some xyz kb that we want to set aside
for
hypervisor]
$free_memory = $total_memory.to_i - (
$mem_allocated_for_running_vms.to_i +
$mem_used_by_base_hypervisor.to_i )
Re: [one-users] Making scheduler allocation aware
The lines
if(scpu==)
{
scpu=0;
}
were accidentally introduced by me in the code. please ignore them.
On 12 November 2010 12:56, Shashank Rachamalla
shashank.rachama...@hexagrid.com wrote:
Hi
As Ruben pointed out, every time a VM is dispatched, host share counter (
mem_usage ) gets incremented. Thus there is no way for opennebula to allow
over committing. However, the problem lies in the following piece of code
from VirtualMachine.cc where cpu, memory and disk are set to 0 when a VM
template does not contain CPU attribute ( this is true in our case ). The
mem_usage value in host share does not get incremented because of this. I
guess, specifying CPU=0 in the template should solve the problem. I will
reconfirm after testing.
void VirtualMachine::get_requirements (int cpu, int memory, int disk)
{
string scpu;
istringstream iss;
float fcpu;
get_template_attribute(MEMORY,memory);
get_template_attribute(CPU,scpu);
*if ((memory == 0) || (scpu==))
{
cpu= 0;
memory = 0;
disk = 0;
return;
}
*
if(scpu==)
{
scpu=0;
}
iss.str(scpu);
iss fcpu;
cpu= (int) (fcpu * 100);//now in 100%
memory = memory * 1024; //now in bytes
disk = 0;
return;
}
On 11 November 2010 19:36, Javier Fontan jfon...@gmail.com wrote:
Hello,
Are you sure that those are the exact values for the host? OpenNebula
counts real (from probes) and allocated (from database) memory so
that should not happen.
Snippet from a onehost show:
--8--
USED MEM (REAL) : 0
USED MEM (ALLOCATED) : 65536
--8--
I am now working on the kvm monitoring and I have noticed another
mismatch even with your probe changes. The values stored in the
database for total memory should be changed and that's what I am
working on.
I am connected to irc.freenode.org in channel #opennebula if you want
to discuss this further.
Bye
On Thu, Nov 11, 2010 at 5:20 AM, Shashank Rachamalla
shashank.rachama...@hexagrid.com wrote:
Hi Javier
Thanks for the inputs but I came across another problem while testing:
If opennebula receives multiple vm requests in a short span of time, the
scheduler might take decisions for all these vms considering the host
monitoring information available from the last monitoring cycle.
Ideally,
before processing every pending request, fresh host monitoring
information
has to be taken into account as the previous set of requests might have
already changed the host’s state. This can result in over committing
when
host is being used close to its full capacity.
Is there any workaround which helps the scheduler to overcome the above
problem ?
steps to reproduce the problem scenario:
Host 1 : Total memory = 3GB
Host 2 : Total memory = 2GB
Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have
a
higher RANK value )
VM1: memory = 2GB
VM2: memroy = 2GB
Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will
come up on Host1. ( Thus over committing )
Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up
on
Host1 and VM2 will come up on Host2. This is true because opennebula
would
have fetched a fresh set of host monitoring information in that time.
On 4 November 2010 02:04, Javier Fontan jfon...@gmail.com wrote:
Hello,
It looks fine to me. I think that taking out the memory the hypervisor
may be consuming is key to make it work.
Bye
On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
rb...@hexagrid.com wrote:
Javier
Yes we are using KVM and OpenNebula 1.4.
We have been having this problem since a long time and we were doing
all
kinds of validations ourselves before submitting the request to
OpenNebula.
(there should be enough memory in the cloud that matches the
requested
memory there should be atleast one host that has memory requested
memory
) We had to do those because OpenNebula would schedule to an
arbitrary
host based on the existing logic it had.
So at last we thought that we need to make OpenNebula aware of memory
allocated of running VM's on the host and started this discussion.
Thanks for taking up this issue as priority. Appreciate it.
Shashank came up with this patch to kvm.rb. Please take a look and
let
us
know if that will work until we get a permanent solution.
$mem_allocated_for_running_vms=0
for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
$dominfo=`virsh dominfo #{i}`
$dominfo.split(/\n/).each{|line|
if line.match('^Max memory')
$mem_allocated_for_running_vms += line.split(
)[2].strip.to_i
end
}
end
