Hash: SHA1

On Monday March 30, 2009, Brian Bouterse wrote:
> While VCL now supports ESX/ESX 3i netboot based virtual machines, the
> VCL architecture presents a lot of real challenges for making this a
> scalable solution, and I'd like to identify and discuss a few points/
> concerns here.
> VCL requires an entry in the computers table for each VM, and this
> entry needs to be tied to a vmhost.  By hard selecting the virtual
> machine entries in the computer table (a VM "slot") up front, the
> decision about where to place the next virtual machine isn't handled
> effectively.  Each VM "slot" gets statically assigned physical
> characteristics at creation time.  This very quickly creates a
> situation where there is space in the datacenter for a particular VM
> on one hypervisor or another, but VCL can't figure it out because the
> large RAM slots got used first for other images, and now the image in
> question can't find a slot to meet it's meta-data requirements.  VCL
> will incorrectly report that there is not space in the infrastructure,
> when there really is.  This is bad.
> Also, it makes the setup much more difficult since an average, modern
> blade can run 20 - 30 VMs, and if you manage an entire blade center
> you're manually creating 350 computer table entries.  Each entry
> requires multiple, manual updates to the database.  This is not a
> tractable solution, and needs to be addressed.

To summarize what you've stated - the scheduler in the web frontend is not 
hypervisor aware.  The reason for this goes way back to a time when Aaron was 
not backlogged in development (on the backend) and I was backlogged (on the 
frontend).  Aaron came up with a way to add in support for vmware as a 
backend only implementation (since I didn't have time to add it in to the 
frontend).  A few months later, we designed out a really good solution for 
dynamically managing hypervisors that would handle all of the virtual machine 
creation in the computer table, automatically adding/removing host servers as 
needed, and reassigning VMs to hosts as needed.  Unfortunately, the tide 
switched and Aaron became swamped, and I had more development time.  I was 
able to implement the frontend part of this (though it never made it into 
production code), but Aaron never had time to implement the backend part.  
Since we had something working that met our needs, other things became a 

Aaron just looked yesterday, and he still has the notes on what we designed 
(back in 2007).  I still have a copy of what I coded around somewhere.  We'll 
see about getting those notes posted online and possibly creating a JIRA 
issue for it to place in our release schedule somewhere.

> As a possible solution (or part of one), one major moving part is the
> placement decision for a particular reservation (tantamount to which
> hypervisor this VM will be reserved on).  Placement today in VCL is
> decided in the front-end without asking the hypervisors what their
> capabilities are of accepting the next VM (or cluster of VMs) in
> question.  One possible way around this is to create a ESX placement
> controller module which determines where to place things.  This module
> would be part of the VCL backend (although it could be called from the
> frontend).  The ESX provisioning module authors each VMX file on the
> fly based on meta-data from the database, so it is already dynamic
> enough.

This concept is tricky.  Currently, there is no mechanism for frontend 
initiated communication directly with the backend.  I agree with your point 
that certain provisioning engines are aware of their own capacity, but can't 
easily keep that capacity in the database such that it is easily accessible 
by the frontend.  However, it's difficult to make a scalable solution where 
there could be many provisioning servers that would have to be quickly polled 
by the frontend before reporting to a user whether or not their request can 
be fulfilled.  I think for the near term, the solution mentioned above will 
meet most of our needs so that we can address this issue of the frontend 
having to get capacity information directly from the backend further down the 

- -- 
- -------------------------------
Josh Thompson
Systems Programmer
Virtual Computing Lab (VCL)
North Carolina State University


my GPG/PGP key can be found at pgp.mit.edu
Version: GnuPG v1.4.6 (GNU/Linux)


Reply via email to