Author: bmbouter
Date: Thu Mar 19 20:32:33 2009
New Revision: 756176
URL: http://svn.apache.org/viewvc?rev=756176&view=rev
Log:
VCL-29
Removed all configuration which required the setting of variables inside the
esx.pm file. These items have now been moved to the database in the vmhost
table.
Removed the requirement that all management nodes mount the storage library via
NFS. These connections have been refactored to use SSH.
Updated the instructions to reflect these changes
Modified:
incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README
incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm
Modified:
incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README
URL:
http://svn.apache.org/viewvc/incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README?rev=756176&r1=756175&r2=756176&view=diff
==============================================================================
--- incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README
(original)
+++ incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.README
Thu Mar 19 20:32:33 2009
@@ -2,13 +2,9 @@
You need to install the VMware VI Perl toolkit installed on any management
node using the esx module
-Each vmware ESX hypervisor requires a mounted NFS datastore named 'VCL'
+Each vmware ESX hypervisor you want VCL to provision to requires the storage
library mounted via NFS and named 'VCL'
-Mount this same datastore on the VCL management node /mnt/vcl/
-
-This NFS export needs to be exported (from the NFS server) at /mnt/export/
-
-Populate this datastore with the following structure
+Ensure your NFS exported storage library has the following structure
[VCL]
/inuse (required)
/golden (required)
@@ -18,9 +14,9 @@
...
Image requirements:
- All images used by this module must have names that begin with
esx3-<anything you want>
+ All images used by this module must have names in the form of
esx3-<anything you want>-v# (where # is the image revision number)
ssh must be set to start on boot
- ssh keys must be setup so that VCL can ssh into it
+ ssh keys must be setup so that VCL can ssh into it (put another way,
the /root/.ssh/id_rsa.pub key from a VCL management node must be added to each
image's authorized_keys file)
The image needs two NICs (eth0 is private, eth1 is public)
The IP addressing of the private (eth0) interface must be set to DHCP
on boot
@@ -30,39 +26,28 @@
max-lease-time 43200;
min-lease-time 43200;
default-lease-time 43200;
- option routers 10.0.0.1;
option subnet-mask 255.255.0.0;
option nis-domain "NA";
- option domain-name "vcl.internal";
- option domain-name-servers 10.0.0.1;
option nis-servers noip;
option time-offset -5;
range 10.0.0.2 10.0.255.255;
-Set a few variables and install the module:
-
-This module expects the following three variables to be hard-coded at the top
of the esx.pm file. Remember to character escape any special characters.
-
-$vmhost_username, $vmhost_password, $datastore_ip
-
-A note about datastore_ip. Because the module (from the management node)
ssh's to the storage system directly instead of doing a cp over nfs, you need
to set the IP of the datastore here. The machine exporting the NFS datastore
should be setup with SSH keys from the VCL ssh keys. This is set in the VCL
conf file as the IDENTITY_blade_linux variable.
-
-Place the module (esx.pm) in your management node's
lib/VCL/Module/Provisioning/ directory
-
-
-
-Do these once:
+Install the module:
-1) Insert a row into the module table with `perlpackage` equal to
'VCL::Module::Provisioning::esx'
-2) Insert a row into the provisioning table with `moduleid` equal to the id
of the entry from step 1 (inserting into the module table)
-3) Insert a record into the 'vmprofile' table with the following guidelines
+1) Place the module (esx.pm) in your management node's
lib/VCL/Module/Provisioning/ directory
+2) Insert a row into the module table with `perlpackage` equal to
'VCL::Module::Provisioning::esx'
+3) Insert a row into the provisioning table with `moduleid` equal to the id
of the entry from step 1 (inserting into the module table)
+4) Insert a record into the 'vmprofile' table with the following guidelines
profilename: VMware ESX SAN
vmtypeid: link to anything valid ID from the vmtype table.
imageid: link to the "No Image" id in the image table
+ datastorepath: <ipaddress>:<NFS mount path> for example a valid entry
could be (152.14.17.112:/mnt/export)
virtualswitch0: <name of your private virtual machine port group>
virtualswitch1: <name of your public virtual machine port group>
+ username: <name of a valid user on your ESX hypervisors>
+ password: <password for the ESX hypervisor user>
@@ -78,7 +63,7 @@
2) For each ESX hypervisor, manually create an entry in the 'vmhost' database
table with the following guidelines:
computerid: the id of the computer created in step 1
vmlimit: the max number of vms to have on this hypervisor (ie: 5)
- vmprofileid: the id of the vmprofile entry created in step 3 of the
above section labeled "Do these once"
+ vmprofileid: the id of the vmprofile entry created in step 4 of the
above section labeled "Install the module"
@@ -97,6 +82,6 @@
1) Create an entry in the image table.
name: esx3-<what you want>-v0
- OSid: link to the fc9 entry on the OS table
-2) Create an image revision table with the imageid fromt the entry created in
(1) and the imagename used in (1)
+ OSid: link to the correct operating system entry in the OS table
+2) Create an image revision table with the imageid from the entry created in
the image table and the imagename used in that same entry
3) Add the image revision to the allImages and allVMimages groups
Modified: incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm
URL:
http://svn.apache.org/viewvc/incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm?rev=756176&r1=756175&r2=756176&view=diff
==============================================================================
--- incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm
(original)
+++ incubator/vcl/trunk/managementnode/lib/VCL/Module/Provisioning/esx.pm Thu
Mar 19 20:32:33 2009
@@ -29,9 +29,12 @@
=head1 DESCRIPTION
- This module provides VCL support for vmware server
+ This module provides VCL support for vmware esx
http://www.vmware.com
+ TODO list:
+ Refactor all run_ssh_command calls to check the return code and fail if not 0
+
=cut
##############################################################################
@@ -156,16 +159,20 @@
my $vmclient_eth1MAC =
$self->data->get_computer_eth1_mac_address;
my $vmclient_OSname = $self->data->get_image_os_name;
- #eventually get these from a config file or database
-
- my $vmhost_username = "vcl";
- my $vmhost_password = "AUqGcDDejbFxN6qX";
- my $datastore_ip = "152.14.17.112";
- my $datastore_share_path = "/mnt/export";
+ my $vmhost_username = $self->data->get_vmhost_profile_username();
+ my $vmhost_password = $self->data->get_vmhost_profile_password();
+
+ #Get the config datastore information from the database
+ my $datastore_ip;
+ my $datastore_share_path;
+ ($datastore_ip, $datastore_share_path) = split(":",
$self->data->get_vmhost_profile_datastore_path());
+ notify($ERRORS{'OK'}, 0, "DATASTORE IP is $datastore_ip and
DATASTORE_SHARE_PATH is $datastore_share_path");
notify($ERRORS{'OK'}, 0, "Entered ESX module, loading $image_name on
$computer_shortname (on $vmhost_hostname) for reservation $reservation_id");
+ notify($ERRORS{'DEBUG'}, 0, "Datastore:
$datastore_ip:$datastore_share_path");
- my $datastorepath4vmx = "/mnt/vcl/inuse/$computer_shortname";
+ # path to the inuse vm folder on the datastore (not a local path)
+ my $vmpath = "$datastore_share_path/inuse/$computer_shortname";
# query the host to see if the vm currently exists
my $vminfo_command = "/usr/lib/vmware-viperl/apps/vm/vminfo.pl";
@@ -208,40 +215,40 @@
notify($ERRORS{'DEBUG'}, 0, "Un-Registered:
$unregister_output");
}
- my $remove_vm_output = `rm -rf $datastorepath4vmx`;
- notify($ERRORS{'DEBUG'}, 0, "Output from remove command is:
$remove_vm_output");
-
- # copy appropriate vmdk file
- my $newdir = $datastorepath4vmx;
- if (!mkdir($newdir)) {
- notify($ERRORS{'CRITICAL'}, 0, "Could not create new directory:
$!");
+
+ # Remove old vm folder
+ run_ssh_command($datastore_ip, $image_identity, "rm -rf $vmpath");
+ notify($ERRORS{'DEBUG'}, 0, "Removed old vm folder");
+
+ # Create new folder for this vm
+ if (!run_ssh_command($datastore_ip, $image_identity, "mkdir $vmpath")) {
+ notify($ERRORS{'CRITICAL'}, 0, "Could not create new
directory");
return 0;
}
- my $from = "/mnt/vcl/golden/$image_name/$image_name.vmdk";
- my $to = "$datastorepath4vmx/$image_name.vmdk";
- if (!copy($from, $to)) {
- notify($ERRORS{'CRITICAL'}, 0, "Could not copy VMDK file! $!");
- # insert load log here perhaps
+
+
+ # copy appropriate vmdk file
+ my $from = "$datastore_share_path/golden/$image_name/$image_name.vmdk";
+ my $to = "$vmpath/$image_name.vmdk";
+ if (!run_ssh_command($datastore_ip, $image_identity, "cp $from $to")) {
+ notify($ERRORS{'CRITICAL'}, 0, "Could not copy vmdk file!");
return 0;
}
notify($ERRORS{'DEBUG'}, 0, "COPIED VMDK SUCCESSFULLY");
# Copy the (large) -flat.vmdk file
- # This uses ssh to do the copy locally on the nfs server.
+ # This uses ssh to do the copy locally, copying over nfs is too costly
$from =
"$datastore_share_path/golden/$image_name/$image_name-flat.vmdk";
- $to =
"$datastore_share_path/inuse/$computer_shortname/$image_name-flat.vmdk";
- my @copy_command = ("ssh", $datastore_ip, "-i", $image_identity, "-o",
"BatchMode yes", "cp $from $to");
- notify($ERRORS{'OK'}, 0, "SSHing to copy vmdk-flat file");
- if (system(@copy_command) >> 8) {
- notify($ERRORS{'CRITICAL'}, 0, "Could not copy VMDK-flat file!
$!");
- # insert load log here perhaps
+ $to = "$vmpath/$image_name-flat.vmdk";
+ if (!run_ssh_command($datastore_ip, $image_identity, "cp $from $to")) {
+ notify($ERRORS{'CRITICAL'}, 0, "Could not copy vmdk-flat
file!");
return 0;
}
- # Author new VMX file
+ # Author new VMX file, output to temporary file (will scp it below)
my @vmxfile;
- my $vmxpath = "$datastorepath4vmx/$image_name.vmx";
+ my $vmxpath = "/tmp/$computer_shortname.vmx";
my $guestOS = "other";
$guestOS = "linux" if ($image_os_name =~ /(fc|centos)/i);
@@ -249,16 +256,19 @@
# determine adapter type by looking at vmdk file
my $adapter = "lsilogic"; # default
- if (open(RE, "grep adapterType $datastorepath4vmx/$image_name.vmdk 2>&1
|")) {
- my @LIST = <RE>;
- close(RE);
- foreach my $a (@LIST) {
- if ($a =~ /(ide|buslogic|lsilogic)/) {
+ my @output;
+ if (@output = run_ssh_command($datastore_ip, $image_identity, "grep
adapterType $vmpath/$image_name.vmdk 2>&1")) {
+ my @LIST = @{$output[1]};
+ foreach (@LIST) {
+ if ($_ =~ /(ide|buslogic|lsilogic)/) {
$adapter = $1;
notify($ERRORS{'OK'}, 0, "adapter= $1 ");
}
}
- } ## end if (open(RE, "grep adapterType
$VMWAREREPOSITORY/$image_name/$image_name.vmdk 2>&1 |"...
+ } else {
+ notify($ERRORS{'CRITICAL'}, 0, "Could not ssh to grep the vmdk
file");
+ return 0;
+ }
push(@vmxfile, "#!/usr/bin/vmware\n");
push(@vmxfile, "config.version = \"8\"\n");
@@ -293,7 +303,7 @@
push(@vmxfile, "scsi0:0.deviceType = \"scsi-hardDisk\"\n");
push(@vmxfile, "scsi0:0.fileName =\"$image_name.vmdk\"\n");
- #write to tmpfile
+ # write vmx to temp file
if (open(TMP, ">$vmxpath")) {
print TMP @vmxfile;
close(TMP);
@@ -305,6 +315,12 @@
return 0;
}
+ # scp $vmxpath to $vmpath/$image_name.vmx
+ if (!run_scp_command($vmxpath, "$datastore_ip:$vmpath/$image_name.vmx",
$image_identity)) {
+ notify($ERRORS{'CRITICAL'}, 0, "could not scp vmx file to
$datastore_ip");
+ return 0;
+ }
+
# Register new vm on host
my $register_command = "/usr/lib/vmware-viperl/apps/vm/vmregister.pl";
$register_command .= " --server '$vmhost_hostname'";
@@ -487,37 +503,43 @@
my $new_imagename = $self->data->get_image_name;
my $computer_shortname = $self->data->get_computer_short_name;
my $image_identity = $self->data->get_image_identity;
- my $vmhost_username = "vcl";
- my $vmhost_password = "AUqGcDDejbFxN6qX";
- my $datastore_ip = "152.14.17.112";
- my $datastore_share_path = "/mnt/export";
+ my $vmhost_username = $self->data->get_vmhost_profile_username();
+ my $vmhost_password = $self->data->get_vmhost_profile_password();
+ #Get the config datastore information from the database
+ my $datastore_ip;
+ my $datastore_share_path;
+ ($datastore_ip, $datastore_share_path) = split(":",
$self->data->get_vmhost_profile_datastore_path());
- my $inuse_image = "/mnt/vcl/inuse/$computer_shortname";
- my $new_golden = "/mnt/vcl/golden/$new_imagename";
+ my $old_vmpath = "$datastore_share_path/inuse/$computer_shortname";
+ my $new_vmpath = "$datastore_share_path/golden/$new_imagename";
+
+ # These four vars are useful:
+ # $old_vmpath, $new_vmpath, $old_imagename, $new_imagename
# Find old image name:
- my $oldimage;
- if (open(LISTFILES, "ls -1 $inuse_image 2>&1 |")) {
- my @list = <LISTFILES>;
- close(LISTFILES);
- my $numfiles = @list;
+ my $old_imagename;
+ #if (open(LISTFILES, "ls -1 $inuse_image 2>&1 |")) {
+ my @ssh_output;
+ if (@ssh_output = run_ssh_command($datastore_ip, $image_identity, "ls
-1 $old_vmpath 2>&1")) {
+ my @list = @{$ssh_output[1]};
#figure out old name
foreach my $a (@list) {
chomp($a);
if ($a =~ /(.*)-(v[0-9]*)\.vmdk/) {
- $oldimage = "$1-$2";
+ $old_imagename = "$1-$2";
}
}
} else {
notify($ERRORS{'CRITICAL'}, 0, "LS failed");
return 0;
}
- notify($ERRORS{'DEBUG'}, 0, "found previous name= $oldimage");
+ notify($ERRORS{'DEBUG'}, 0, "found previous name= $old_imagename");
notify($ERRORS{'OK'}, 0, "SSHing to node to configure
currentimage.txt");
- my @sshcmd = run_ssh_command($computer_shortname, $image_identity,
"echo $new_imagename > /root/currentimage.txt", "root");
+ # XXX SHOULD INSTEAD USE write_currentimage_txt IN utils.pm
+ my @sshcmd = run_ssh_command($computer_shortname, $image_identity,
"echo $new_imagename > /root/currentimage.txt");
my $poweroff_command = "/usr/lib/vmware-viperl/apps/vm/vmcontrol.pl";
$poweroff_command .= " --server '$vmhost_hostname'";
@@ -532,49 +554,50 @@
notify($ERRORS{'OK'}, 0, "Waiting 5 seconds for power off");
sleep(5);
- # copy appropriate vmdk file
- my $newdir = $new_golden;
- if (!mkdir($newdir)) {
+
+ # Make the new golden directory
+ if (!run_ssh_command($datastore_ip, $image_identity, "mkdir
$new_vmpath")) {
notify($ERRORS{'CRITICAL'}, 0, "Could not create new directory:
$!");
return 0;
}
- my $from = "$inuse_image/$oldimage.vmdk";
- my $to = "$new_golden/$new_imagename.vmdk";
- notify($ERRORS{'DEBUG'}, 0, "Preparing to copy vmdk from $from to $to");
- if (!copy($from, $to)) {
+
+ # copy appropriate vmdk file
+ my $from = "$old_vmpath/$old_imagename.vmdk";
+ my $to = "$new_vmpath/$new_imagename.vmdk";
+ if (!run_ssh_command($datastore_ip, $image_identity, "cp $from $to")) {
notify($ERRORS{'CRITICAL'}, 0, "Could not copy VMDK file! $!");
- # insert load log here perhaps
return 0;
}
notify($ERRORS{'DEBUG'}, 0, "COPIED VMDK SUCCESSFULLY");
- $from = "$inuse_image/$oldimage.vmx";
- $to = "$new_golden/$new_imagename.vmx";
- notify($ERRORS{'DEBUG'}, 0, "Preparing to copy vmx from $from to $to");
- if (!copy($from, $to)) {
+ # Now copy the vmx file (for debugging, vmx isn't actually used. This
code can be taken out)
+ $from = "$old_vmpath/$old_imagename.vmx";
+ $to = "$new_vmpath/$new_imagename.vmx";
+ if (!run_ssh_command($datastore_ip, $image_identity, "cp $from $to")) {
notify($ERRORS{'CRITICAL'}, 0, "Could not copy VMX file! $!");
- # insert load log here perhaps
return 0;
}
notify($ERRORS{'DEBUG'}, 0, "COPIED VMX SUCCESSFULLY");
my $output;
notify($ERRORS{'OK'}, 0, "Rewriting VMDK and VMX files with new image
name");
- $output = `sed -i 's/$oldimage/$new_imagename/'
$new_golden/$new_imagename.vmx`;
- notify($ERRORS{'DEBUG'}, 0, "VMX sed: $output");
- $output = `sed -i 's/$oldimage/$new_imagename/'
$new_golden/$new_imagename.vmdk`;
- notify($ERRORS{'DEBUG'}, 0, "VMDK sed: $output");
+ if (!run_ssh_command($datastore_ip, $image_identity, "sed -i
\"s/$old_imagename/$new_imagename/\" $new_vmpath/$new_imagename.vmx")){
+ notify($ERRORS{'CRITICAL'}, 0, "Sed error");
+ return 0;
+ }
+ if (!run_ssh_command($datastore_ip, $image_identity, "sed -i
\"s/$old_imagename/$new_imagename/\" $new_vmpath/$new_imagename.vmdk")){
+ notify($ERRORS{'CRITICAL'}, 0, "Sed error");
+ return 0;
+ }
# Copy the (large) -flat.vmdk file
# This uses ssh to do the copy locally on the nfs server.
- $from =
"$datastore_share_path/inuse/$computer_shortname/$oldimage-flat.vmdk";
- $to =
"$datastore_share_path/golden/$new_imagename/$new_imagename-flat.vmdk";
+ $from = "$old_vmpath/$old_imagename-flat.vmdk";
+ $to = "$new_vmpath/$new_imagename-flat.vmdk";
notify($ERRORS{'DEBUG'}, 0, "Preparing to ssh to $datastore_ip copy
vmdk-flat from $from to $to");
- my @copy_command = ("ssh", $datastore_ip, "-i", $image_identity, "-o",
"BatchMode yes", "cp $from $to");
notify($ERRORS{'OK'}, 0, "SSHing to copy vmdk-flat file");
- if (system(@copy_command) >> 8) {
- notify($ERRORS{'CRITICAL'}, 0, "Could not copy VMDK-flat file!
$!");
- # insert load log here perhaps
+ if (!run_ssh_command($datastore_ip, $image_identity, "cp $from $to")) {
+ notify($ERRORS{'CRITICAL'}, 0, "Could not copy VMDK-flat
file!");
return 0;
}
@@ -725,26 +748,38 @@
return 0;
}
+ my $image_identity = $self->data->get_image_identity;
my $image_name = $self->data->get_image_name();
+
+ #Get the config datastore information from the database
+ my $datastore_ip;
+ my $datastore_share_path;
+ ($datastore_ip, $datastore_share_path) = split(":",
$self->data->get_vmhost_profile_datastore_path());
+
if (!$image_name) {
notify($ERRORS{'CRITICAL'}, 0, "unable to determine if image
exists, unable to determine image name");
return 0;
}
- my $IMAGEREPOSITORY = "/mnt/vcl/golden";
+ my $goldenpath = "$datastore_share_path/golden";
- if (open(IMAGES, "/bin/ls -1 $IMAGEREPOSITORY 2>&1 |")) {
- my @images = <IMAGES>;
- close(IMAGES);
- foreach my $i (@images) {
- if ($i =~ /$image_name/) {
+ my @ssh_output;
+ if (@ssh_output = run_ssh_command($datastore_ip, $image_identity, "ls
-1 $goldenpath 2>&1")) {
+ my @list = @{$ssh_output[1]};
+ #figure out old name
+ foreach my $a (@list) {
+ chomp($a);
+ if ($a =~ /$image_name/) {
notify($ERRORS{'OK'}, 0, "image $image_name
exists");
return 1;
}
}
- } ## end if (open(IMAGES, "/bin/ls -1 $IMAGEREPOSITORY 2>&1 |"...
+ } else {
+ notify($ERRORS{'CRITICAL'}, 0, "LS failed");
+ return 0;
+ }
- notify($ERRORS{'WARNING'}, 0, "image $IMAGEREPOSITORY/$image_name does
NOT exists");
+ notify($ERRORS{'WARNING'}, 0, "image $goldenpath/$image_name does NOT
exists");
return 0;
} ## end sub does_image_exist
@@ -768,6 +803,7 @@
# Either use a passed parameter as the image name or use the one stored
in this object's DataStructure
my $image_name = shift;
+ my $image_identity = $self->data->get_image_identity;
$image_name = $self->data->get_image_name() if !$image_name;
if (!$image_name) {
notify($ERRORS{'CRITICAL'}, 0, "image name could not be
determined");
@@ -775,12 +811,17 @@
}
notify($ERRORS{'DEBUG'}, 0, "getting size of image: $image_name");
- my $IMAGEREPOSITORY = "/mnt/vcl/golden/$image_name";
+ #Get the config datastore information from the database
+ my $datastore_ip;
+ my $datastore_share_path;
+ ($datastore_ip, $datastore_share_path) = split(":",
$self->data->get_vmhost_profile_datastore_path());
+
+ my $IMAGEREPOSITORY = "$datastore_share_path/golden/$image_name";
#list files in image directory, account for main .gz file and any
.gz.00X files
- if (open(FILELIST, "/bin/ls -s1 $IMAGEREPOSITORY 2>&1 |")) {
- my @filelist = <FILELIST>;
- close(FILELIST);
+ my @output;
+ if (@output = run_ssh_command($datastore_ip, $image_identity, "/bin/ls
-s1 $IMAGEREPOSITORY 2>&1")) {
+ my @filelist = @{$output[1]};
my $size = 0;
foreach my $f (@filelist) {
if ($f =~ /$image_name-flat.vmdk/) {
@@ -793,7 +834,7 @@
return 0;
}
return int($size / 1024);
- } ## end if (open(FILELIST, "/bin/ls -s1 $IMAGEREPOSITORY 2>&1 |"...
+ }
return 0;
} ## end sub get_image_size
