Author: bmbouter
Date: Sun Feb 1 21:12:28 2009
New Revision: 739839
URL: http://svn.apache.org/viewvc?rev=739839&view=rev
Log:
VCL-29
This patch reworks how the module discovers the private IP address of VMs that
it provisions to ESX and ESX 3i hosts. Specifically, it implements the
following new work flow:
1) Create the VM on the hypervisor, but DO NOT specify any mac addresses
2) Power on the VM
3) Ask the hypervisor, what the MAC address is of the eth0 (private) interface
of the vm just provisioned
4) Watch the arp table of the management module for the dhcp address handed
back to the VM when it's eth0 does the required dhcp request.
5) Delete the old entry from /etc/hosts for the VM
6) Add the new entry to /etc/hosts so that VCL can resolve this VM's hostname
for account setup, etc...
This simplifies the setup significantly by removing the following three manual,
per-vm steps:
1) Manual setting of eth0 mac address in database
2) Manual configuration of that same eth0 mac in the the management module's
dhcpd.conf
3) Manual addition of IP/hostname pair to /etc/hosts file
The documentation (esx.README) has been simplified and updated as a result
Modified:
incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README
incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm
Modified:
incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README
URL:
http://svn.apache.org/viewvc/incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README?rev=739839&r1=739838&r2=739839&view=diff
==============================================================================
--- incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README
(original)
+++ incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README
Sun Feb 1 21:12:28 2009
@@ -1,19 +1,18 @@
Pre-Installation:
-You need to install the VMware VI Perl toolkit installed on the VCL server
+You need to install the VMware VI Perl toolkit installed on any management
node using the esx module
-Each vmware esx hypervisor requires a mounted NFS datastore named 'VCL'
+Each vmware ESX hypervisor requires a mounted NFS datastore named 'VCL'
-This same datastore needs to be mounted on the VCL server at /mnt/vcl/
+Mount this same datastore on the VCL management node /mnt/vcl/
This NFS export needs to be exported (from the NFS server) at /mnt/export/
-This same datastore requires the following structure.
-
+Populate this datastore with the following structure
[VCL]
/inuse (required)
/golden (required)
- /imagename
+ /imagename (a folder in golden for each image in the library)
imagename.vmdk
imagename-flat.vmdk
...
@@ -23,25 +22,41 @@
ssh must be set to start on boot
ssh keys must be setup so that VCL can ssh into it
The image needs two NICs (eth0 is private, eth1 is public)
- The IP addressing of the private (eth1) interface must be set to DHCP
on boot
+ The IP addressing of the private (eth0) interface must be set to DHCP
on boot
+
+The VCL management node acts as a dhcp server for the private (eth0) NICs of
the virtual machines that will be provisioned. Configure your management
node's private interface eth1 (10.0.0.1) to hand out valid IP addresses on the
network 10.0.0.0/16 by configuring your /etc/dhcpd.conf and turning on dhcp.
Use the following entry as a guide:
+ subnet 10.0.0.0 netmask 255.255.0.0 {
+ max-lease-time 43200;
+ min-lease-time 43200;
+ default-lease-time 43200;
+ option routers 10.0.0.1;
+ option subnet-mask 255.255.0.0;
+ option nis-domain "NA";
+ option domain-name "vcl.internal";
+ option domain-name-servers 10.0.0.1;
+ option nis-servers noip;
+ option time-offset -5;
+ range 10.0.0.2 10.0.255.255;
-VCL expects the blades to have a consistant private IP address. Go configure
dhcpd.conf and in /etc/hosts manually
-Customize the module:
+Set a few variables and install the module:
This module expects the following three variables to be hard-coded at the top
of the esx.pm file. Remember to character escape any special characters.
$vmhost_username, $vmhost_password, $datastore_ip
-A note about datastore_ip. Because the module ssh's to the storage system
directly instead of doing a cp over nfs, you need to set the IP of the
datastore here. The datastore machine should be setup with SSH keys from the
VCL ssh keys set as IDENTITY_blade_linux in the VCL conf file.
+A note about datastore_ip. Because the module (from the management node)
ssh's to the storage system directly instead of doing a cp over nfs, you need
to set the IP of the datastore here. The machine exporting the NFS datastore
should be setup with SSH keys from the VCL ssh keys. This is set in the VCL
conf file as the IDENTITY_blade_linux variable.
+
+Place the module (esx.pm) in your management node's
lib/VCL/Module/Provisioning/ directory
+
-Install the module:
Do these once:
+
1) Insert a row into the module table with `perlpackage` equal to
'VCL::Module::Provisioning::esx'
-2) Insert a row into the provisioning table with `moduleid` equal to the id
of the entry from step 1 in the module table
+2) Insert a row into the provisioning table with `moduleid` equal to the id
of the entry from step 1 (inserting into the module table)
3) Insert a record into the 'vmprofile' table with the following guidelines
profilename: VMware ESX SAN
vmtypeid: link to anything valid ID from the vmtype table.
@@ -49,38 +64,39 @@
virtualswitch0: <name of your private virtual machine port group>
virtualswitch1: <name of your public virtual machine port group>
-Make VCL aware of your ESX hosts:
-1) For each esx hypervisor, which VCL should provisong virtual machine to,
create an entry for that hypervisor by going to Manage Computers -> Edit
Computer Information -> Add
+
+
+For each ESX hypervisor which VCL will provision machines on, do the following:
+
+1) Create an entry for that hypervisor by going to Manage Computers -> Edit
Computer Information -> Add
Hostname <your ESX's hostname>
IP <your ESX's IP>
Type "blade"
Provisioning Engine: This doesn't matter
Computer Groups: Don't add the hypervisors to any computer groups
-2) For each esx hypervisor manually create an entry in the 'vmhost' database
table with the following guidelines:
+2) For each ESX hypervisor, manually create an entry in the 'vmhost' database
table with the following guidelines:
computerid: the id of the computer created in step 1
- vmlimit: the max number of vms to have on this hypervisor
- vmprofileid: the id of the vmprofile entry created in the "Do these
once" secion step 3
+ vmlimit: the max number of vms to have on this hypervisor (ie: 5)
+ vmprofileid: the id of the vmprofile entry created in step 3 of the
above section labeled "Do these once"
+
+
+
+Assigning VMs to your hypervisors:
-Create a virtual machine placeholder for each VM you would like to
concurrently run. Do so by going to Manage Computers -> Edit Computer
Information -> Add
+1) Create a virtual machine placeholder for each VM you would like to
concurrently run. Do so by going to Manage Computers -> Edit Computer
Information -> Add
Hostname: <the hostname of the vm when it is provisioned and turned on>
- IP: <the public IP of the vm when it is provisioned and turned on>
+ IP: this doesn't matter, our module will populate this field
Type: virtualmachine
Provisioning Engine: VMware ESX
Computer Groups: allComputers
-For each virtual machine, manually update the private IP, eth0macaddress
(private), and eth1macaddress (public) for the placeholder virtual machine
entries created in in the computers table during step (4)
-
-Add static entries in dhcpd.conf for the eth0macaddresses (private) for all
computer placeholder entries
-
-Add similar entries for each compter placeholder to the /etc/hosts file to
provide VCL with name resolution on these vms
-
-Finally, once all virtual machines placeholders have been created, 'assign'
them to hypervisors by using the 'Virtual Hosts' area of your web interface.
+2) 'assign' these computer placeholders (and their hostnames) to hypervisors
by using the 'Virtual Hosts' area of your web interface.
-Now your hypervisors and virtual machines are setup, time to add an image:
+Now your hypervisors and virtual machines are setup. Add an image:
1) Create an entry in the image table.
name: esx3-<what you want>-v0
OSid: link to the fc9 entry on the OS table
2) Create an image revision table with the imageid fromt the entry created in
(1) and the imagename used in (1)
-3) Add the image revision to the allImages and allVMimages groups
\ No newline at end of file
+3) Add the image revision to the allImages and allVMimages groups
Modified: incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm
URL:
http://svn.apache.org/viewvc/incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm?rev=739839&r1=739838&r2=739839&view=diff
==============================================================================
--- incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm
(original)
+++ incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm Sun
Feb 1 21:12:28 2009
@@ -60,6 +60,12 @@
use VCL::utils;
use Fcntl qw(:DEFAULT :flock);
+# Used to query for the MAC address once a host has been registered
+use VMware::VIRuntime;
+use VMware::VILib;
+
+
+
##############################################################################
=head1 CLASS ATTRIBUTES
@@ -150,9 +156,10 @@
my $vmclient_OSname = $self->data->get_image_os_name;
#eventually get these from a config file or database
- my $vmhost_username = "";
- my $vmhost_password = "";
- my $datastore_ip = "";
+
+ my $vmhost_username = "root";
+ my $vmhost_password = "m\...@gecl0ud";
+ my $datastore_ip = "152.14.17.78";
notify($ERRORS{'OK'}, 0, "Entered ESX module, loading $image_name on
$computer_shortname (on $vmhost_hostname) for reservation $reservation_id");
@@ -266,9 +273,9 @@
push(@vmxfile, "ethernet0.wakeOnPcktRcv = \"false\"\n");
push(@vmxfile, "ethernet1.wakeOnPcktRcv = \"false\"\n");
- push(@vmxfile, "ethernet0.address = \"$vmclient_eth0MAC\"\n");
- push(@vmxfile, "ethernet1.address = \"$vmclient_eth1MAC\"\n");
- push(@vmxfile, "ethernet0.addressType = \"static\"\n");
+ #push(@vmxfile, "ethernet0.address = \"$vmclient_eth0MAC\"\n");
+ #push(@vmxfile, "ethernet1.address = \"$vmclient_eth1MAC\"\n");
+ push(@vmxfile, "ethernet0.addressType = \"generated\"\n");
push(@vmxfile, "ethernet1.addressType = \"generated\"\n");
push(@vmxfile, "gui.exitOnCLIHLT = \"FALSE\"\n");
push(@vmxfile, "snapshot.disabled = \"TRUE\"\n");
@@ -324,11 +331,71 @@
notify($ERRORS{'DEBUG'},0,"Powered on: $poweron_output");
- # Start waiting for SSH to come up
+ # Query the VI Perl toolkit for the mac address of our newly registered
+ # machine
+ Vim::login(service_url => "https://$vmhost_hostname/sdk";, user_name =>
$vmhost_username, password => $vmhost_password);
+ my $vm_view = Vim::find_entity_view(view_type => 'VirtualMachine',
filter => {'config.name' => "$computer_shortname"});
+ if (!$vm_view) {
+ notify($ERRORS{'CRITICAL'}, 0, "Could not query for VM in VI
PERL API");
+ Vim::logout();
+ return 0;
+ }
+ my $devices = $vm_view->config->hardware->device;
+ my $mac_addr;
+ foreach my $dev (@$devices) {
+ next unless ($dev->isa ("VirtualEthernetCard"));
+ notify($ERRORS{'DEBUG'}, 0, "deviceinfo->summary:
$dev->deviceinfo->summary");
+ notify($ERRORS{'DEBUG'}, 0, "virtualswitch0: $virtualswitch0");
+ if ($dev->deviceInfo->summary eq $virtualswitch0) {
+ $mac_addr = $dev->macAddress;
+ }
+ }
+ Vim::logout();
+ if (!$mac_addr) {
+ notify($ERRORS{'CRITICAL'}, 0, "Failed to find MAC address");
+ return 0;
+ }
+ notify($ERRORS{'OK'}, 0, "Queried MAC address is $mac_addr");
+
+ # Query ARP table for $mac_addr to find the IP (waiting for machine to
come up if necessary)
+ # The DHCP negotiation should add the appropriate ARP entry for us
+ my $arpstatus = 0;
+ my $wait_loops = 0;
+ my $client_ip;
+ while (!$arpstatus) {
+ my $arpoutput = `arp -n`;
+ if ($arpoutput =~
/^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*?$mac_addr/mi) {
+ $client_ip = $1;
+ $arpstatus = 1;
+ notify($ERRORS{'OK'}, 0, "$computer_shortname now has
ip $client_ip");
+ }
+ else {
+ if ($wait_loops > 24) {
+ notify($ERRORS{'CRITICAL'}, 0, "waited
acceptable amount of time for dhcp, please check $computer_shortname on
$vmhost_hostname");
+ return 0;
+ }
+ else {
+ $wait_loops++;
+ notify($ERRORS{'OK'}, 0, "going to sleep 5
seconds, waiting for computer to DHCP. Try $wait_loops");
+ sleep 5;
+ }
+ }
+ }
+
+
+ notify($ERRORS{'OK'}, 0, "Found IP address $client_ip");
+
+ # Delete existing entry for $computer_shortname in /etc/hosts (if any)
+ notify($ERRORS{'OK'}, 0, "Removing old hosts entry");
+ my $sedoutput = `sed -i "/.*\\b$computer_shortname\$/d" /etc/hosts`;
+ notify($ERRORS{'DEBUG'}, 0, $sedoutput);
+ # Add new entry to /etc/hosts for $computer_shortname
+ `echo -e "$client_ip\t$computer_shortname" >> /etc/hosts`;
+ # Start waiting for SSH to come up
my $sshdstatus = 0;
- my $wait_loops = 0;
+ $wait_loops = 0;
my $sshd_status = "off";
notify($ERRORS{'DEBUG'}, 0, "Waiting for ssh to come up on
$computer_shortname");
while (!$sshdstatus) {
@@ -347,7 +414,7 @@
else {
$wait_loops++;
# to give post config a chance
- notify($ERRORS{'OK'}, 0, "going to sleep 5
seconds, waiting for post config to finish. Try $wait_loops");
+ notify($ERRORS{'OK'}, 0, "going to sleep 5
seconds, waiting for computer to start SSH. Try $wait_loops");
sleep 5;
}
} # else
