Thank you for the comprehensive response Andy!
I will be trying this out and report back later on how it works. This is a huge
help!
-----------------------
Jonathan Casco
HPC System Administrator
Florida International University
From: Andy Kurth <andy_ku...@ncsu.edu>
Reply-To: "user@vcl.apache.org" <user@vcl.apache.org>
Date: Thursday, February 15, 2018 at 1:02 PM
To: "user@vcl.apache.org" <user@vcl.apache.org>
Subject: Re: Question on xCAT implementation in VCL
You've probably already done this, but run:
copycds VMware-VMvisor-Installer-201701001-4887370.x86_64.iso
This should have created:
/install/esxi6.5/x86_64/
You're correct that recent xCAT doesn't seem to create an osimage automatically
by running copycds. To create an osimage manually, first create a stanza file:
vi /root/esxi6.5-x86_64-install-hypervisor.stanza
Add this:
>>> BEGIN
# <xCAT data object stanza file>
esxi6.5-x86_64-install-hypervisor:
objtype=osimage
imagetype=linux
osarch=x86_64
osname=Linux
osvers=esxi6.5
otherpkgdir=/install/post/otherpkgs/esxi6.5/x86_64
profile=hypervisor
provmethod=install
template=/opt/xcat/share/xcat/install/esxi/hypervisor.tmpl
<<< END
Create the osimage from the stanza file:
cat /root/esxi6.5-x86_64-install-hypervisor.stanza | mkdef -z
View the osimage settings:
lsdef -t osimage esxi6.5-x86_64-install-hypervisor
If you want to change any the osimage settings, do so like this:
chdef -t osimage esxi6.5-x86_64-install-hypervisor
template=/opt/xcat/share/xcat/install/esxi/<MY CUSTOM>.tmpl
The template file used will be the stock one. Make sure it exists:
/opt/xcat/share/xcat/install/esxi/hypervisor.tmpl
You need to make sure all of the xCAT node/osimage/etc properties used in the
template are defined. If not, you'll get an error when you run the nodeset
command later on.
For example, the stock template sets the ESXi root password via this line in
hypervisor.tmpl:
rootpw --iscrypted #CRYPT:passwd:key=vmware,username=root:password#
I ran a quick test installing a VM with ESXi 6.5 via xCAT and it seems the
password behavior changed with 6.5. I had to use the following to set an
encrypted password in xCAT. (Plain text in xCAT's passwd table used to work)
Without the openssh part, the installation failed with an error related to a
bad "crypted password":
chtab key=vmware passwd.username=root passwd.password=`openssl passwd -1 '<MY
$TRONG PASSWORD>'`
You'll need to add a node in xCAT, set up the network table, get DHCP working,
and a few other things. These are pretty much stock xCAT procedures so I won't
get into them. Reply if you need help though.
Set the node to use the osimage:
nodeset <NODE> osimage=esxi6.5-x86_64-install-hypervisor
Again, if you get errors it probably means one or more of the xCAT settings are
missing for the node or network.
Before trying to install, verify node's PXE files look correct:
/tftpboot/xcat/xnba/nodes/<NODE>
/tftpboot/xcat/xnba/nodes/<NODE>.pxelinux
/tftpboot/xcat/xnba/nodes/<NODE>.uefi
The main file that will need tweaking is the template file generated for the
node by nodeset (derived from
/opt/xcat/share/xcat/install/esxi/hypervisor.tmpl):
/install/autoinst/<NODE>
If you get the node to install and want to work on tweaking the config, what I
usually do is edit the node's template file directly and then restart/PXE
boot/reinstall it. I don't rerun nodeset in between or else the node's
template file will be overwritten with the osimage template settings.
Once I have the configuration correct, I'll rework the osimage template using
the appropriate xCAT variable substitutions.
Below is an old custom template file used for ESXi 6.0. It certainly needs
some tweaking but may give you some pointers on how to configure things. The
commented out lines at the beginning are used for debugging problems. Even
though commented out, the xCAT variables get substituted when you run nodeset.
I'll put in lines for most of the variables used later on to make it easy to
see if a value is missing or incorrect. After ESXi is installed, we'll run
some custom scripts stored on a NAS datastore which configure the firewall and
do some other tweaks not stored in the template.
>>> BEGIN
# node name: '#TABLE:nodetype:$NODE:node#'
# node private IP address: '#TABLE:hosts:$NODE:ip#'
# node private network: '#TABLE:networks:netname=private:net#'
# node private subnet mask: '#TABLE:networks:netname=private:mask#'
# node private MAC address: '#TABLE:mac:$NODE:mac#'
# nameservers: '#TABLE:site:key=nameservers:value#'
# domain: '#TABLE:site:key=domain:value#'
# management node: '#TABLE:site:key=master:value#'
# TFTP server: '#TABLE:noderes:$NODE:tftpserver#'
# default gateway: '#COMMAND: route -n |grep '^0\.0\.0\.0' |awk '{print $2}' #'
# Install NIC: 'vmnic#COMMAND: echo -n #TABLE:noderes:$NODE:installnic# | grep
-o "[0-9]"#'
# Primary NIC: 'vmnic#COMMAND: echo -n #TABLE:noderes:$NODE:primarynic# | grep
-o "[0-9]"#'
vmaccepteula
rootpw --iscrypted #CRYPT:passwd:key=vmware,username=root:password#
clearpart --firstdisk=USB-IBM,usb,mptsas,mpt2sas,local --overwritevmfs
install --firstdisk=USB-IBM,usb,mptsas,mpt2sas,local --overwritevmfs
serialnum --esx=#TABLE:prodkey:$NODE,product=esxi6.free:key#
network --bootproto=dhcp --device=vmnic#COMMAND: echo -n
#TABLE:noderes:$NODE:installnic# | grep -o "[0-9]"# --addvmportgroup=0
reboot
#
=================================================================================================
%pre --interpreter=busybox
#
=================================================================================================
%post --interpreter=busybox --ignorefailure=true
localcli network firewall unload
sleep 5
echo -e "<xcatrequest>\n<command>nextdestiny</command>\n</xcatrequest>" |
/bin/openssl s_client -quiet -connect #TABLE:site:key=master:value#:3001 2>&1 |
tee /tmp/foo.log
#
=================================================================================================
%firstboot --interpreter=busybox
#esxcfg-advcfg -s 1 /Net/FollowHardwareMac
#sleep 60
INSTALL_NIC=vmnic$(echo -n #TABLE:noderes:$NODE:installnic# | grep -o "[0-9]")
PRIMARY_NIC=vmnic$(echo -n #TABLE:noderes:$NODE:primarynic# | grep -o "[0-9]")
INSTALL_MAC=$(esxcli network nic list | grep $INSTALL_NIC | grep -E -o
'([0-9a-f]{2}:){5}[0-9a-f]{2}')
PRIMARY_MAC=$(esxcli network nic list | grep $PRIMARY_NIC | grep -E -o
'([0-9a-f]{2}:){5}[0-9a-f]{2}')
PRIVATE_IP=#TABLE:hosts:$NODE:ip#
PRIVATE_MASK=#TABLE:networks:netname=private:mask#
echo "INSTALL_NIC: ${INSTALL_NIC}"
echo "INSTALL_MAC: ${INSTALL_MAC}"
echo "PRIMARY_NIC: ${PRIMARY_NIC}"
echo "PRIMARY_MAC: ${PRIMARY_MAC}"
#----------------------------
#esxcli network vswitch standard remove --vswitch-name=vSwitchUSB0
esxcli network ip interface remove --interface-name vmk0
esxcli network vswitch standard portgroup remove --portgroup-name "Management
Network" --vswitch-name vSwitch0
esxcli network vswitch standard add --vswitch-name vSwitch1
esxcli network vswitch standard uplink add --uplink-name ${PRIMARY_NIC}
--vswitch-name vSwitch1
esxcli network vswitch standard portgroup add --portgroup-name VMkernel-Public
--vswitch-name vSwitch1
esxcli network ip interface add --interface-name vmk2 --portgroup-name
VMkernel-Public --mac-address=${PRIMARY_MAC}
esxcli network ip interface ipv4 set --interface-name vmk2 --type dhcp
esxcli network ip interface tag add -i vmk2 -t Management
esxcli network vswitch standard portgroup add --portgroup-name Public
--vswitch-name vSwitch1
esxcli network vswitch standard portgroup add --portgroup-name VMkernel-Private
--vswitch-name vSwitch0
esxcli network ip interface add --interface-name vmk0 --portgroup-name
VMkernel-Private --mac-address=${INSTALL_MAC}
esxcli network ip interface ipv4 set --interface-name vmk0 --ipv4 ${PRIVATE_IP}
--netmask ${PRIVATE_MASK} --type static
esxcli network ip interface tag remove --interface-name vmk0 --tagname
Management
esxcli network vswitch standard portgroup add --portgroup-name Private
--vswitch-name vSwitch0
esxcfg-route esxcfg-route -a default #COMMAND: route -n |grep '^0\.0\.0\.0'
|awk '{print $2}' #
esxcli network ip interface ipv4 set --interface-name vmk2 --peer-dns true
--type=dhcp
esxcli system hostname set --host=#TABLE:nodetype:$NODE:node#
esxcli system hostname set --fqdn=
#TABLE:nodetype:$NODE:node#.#TABLE:site:key=domain:value#
#-----------------------------------------------------------------------------------------------------------
# SSH
esxcli network firewall ruleset set --ruleset-id sshServer --enabled yes
chkconfig SSH on
echo "#COMMAND: cat /etc/vcl/vcl.key.pub#" > /etc/ssh/keys-root/authorized_keys
#-----------------------------------------------------------------------------------------------------------
# STORAGE
LOCAL_DATASTORE_NAME="local_#TABLE:nodetype:$NODE:node#"
REPOSITORY_NAME="repository"
# Rename the local datastore
vim-cmd hostsvc/datastore/rename datastore1 ${LOCAL_DATASTORE_NAME}
vim-cmd hostsvc/datastore/refresh ${LOCAL_DATASTORE_NAME}
vim-cmd hostsvc/storage/refresh
# Mount repository
esxcfg-nas --add --host <NAS IP> --share /repository ${REPOSITORY_NAME}
#-----------------------------------------------------------------------------------------------------------
# Add hostname to terminal prompt
echo "PS1='#TABLE:nodetype:$NODE:node#:\w \$ '" >> /etc/profile.local
#-----------------------------------------------------------------------------------------------------------
# FIREWALL
/vmfs/volumes/${REPOSITORY_NAME}/_scripts/esxi_firewall.sh >
/var/log/esxi_firewall.log
#-----------------------------------------------------------------------------------------------------------
# TWEAKS
/vmfs/volumes/${REPOSITORY_NAME}/_scripts/esxi_tweaks.sh >
/var/log/esxi_tweaks.log
#-----------------------------------------------------------------------------------------------------------
# SAVE LOGS
# Copy %firstboot script logs to persisted datastore
mkdir -p /vmfs/volumes/${LOCAL_DATASTORE_NAME}/install_logs
cp /var/log/* /vmfs/volumes/${LOCAL_DATASTORE_NAME}/install_logs/
#-----------------------------------------------------------------------------------------------------------
# Backup ESXi configuration to persist changes
/sbin/auto-backup.sh
reboot
<<<END
On Wed, Feb 14, 2018 at 2:25 PM, Jonathan Casco
<jca...@fiu.edu<mailto:jca...@fiu.edu>> wrote:
Thanks for that information Andy.
My current hangup is this part of the process in your list:
* Add the computer to xCAT and get things configured so xCAT can install and
properly configure it (this is no easy feat)
I was thinking (as you mentioned) that xCAT would make a VMware host which
could then be used by VCL, likely with the help of a kickstart file.
However xCAT does not seem to be too happy when loading ESXi 6 as it does not
create an entry in the osimage table for it. It seems like this is a feature
which was abandoned in xCAT several versions ago as it was not requested often.
https://github.com/xcat2/xcat-core/issues/4108<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xcat2_xcat-2Dcore_issues_4108&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=0Mz5KaUjFlgb7ODSsLhATg&m=fi8Y17kZ-dX6IpkeLBgX96yFIr0tuKRhqP50_x38gGg&s=S3BcK5Rj8zDw3KQEdtzw0scrWZKcKzdoYRPk0sb3Tao&e=>
Would you be willing to give some detail on what version of ESXi you used in
the past and how you got the xCAT provisioning for ESXi accomplished?
Thank you again,
-----------------------
Jonathan Casco
HPC System Administrator
Florida International University
From: Andy Kurth <andy_ku...@ncsu.edu<mailto:andy_ku...@ncsu.edu>>
Reply-To: "user@vcl.apache.org<mailto:user@vcl.apache.org>"
<user@vcl.apache.org<mailto:user@vcl.apache.org>>
Date: Wednesday, February 14, 2018 at 2:10 PM
To: "user@vcl.apache.org<mailto:user@vcl.apache.org>"
<user@vcl.apache.org<mailto:user@vcl.apache.org>>
Subject: Re: Question on xCAT implementation in VCL
xCAT is only used within VCL to deploy bare-metal machines. The VM deployment
functionality of xCAT isn't used at all. VCL can deploy bare metal machines
for user reservations. It's pretty rare nowadays to deploy end user
reservations on full bare-metal machines, but it's still useful.
For larger installations, you could set things up so that xCAT deploys and
configures the bare-metal VM hosts (VMware and/or KVM). Here at NCSU, we have
some rather complex Kickstart templates for each that set up ESXi, KVM
bridging, etc.
In theory, the flow would go like this:
* Add the bare-metal computer to VCL, set the provisioning to xCAT
* Add the computer to xCAT and get things configured so xCAT can install and
properly configure it (this is no easy feat)
* Add an image to the VCL database matching the xCAT template name
* In VCL, change the computer's state by selecting "convert to vmhostinuse"
* VCL instructs xCAT to automatically reload the computer using the xCAT
template name
* When done, the computer is in the "vmhostinuse" state and you can assign VMs
to it
In practice, I gave up on having VCL initiate the xCAT deployment long ago.
There are too many things that can break. It's easier to initiate the xCAT
reinstall manually, verify, then add the computer to VCL with the provisioning
module set to none so that VCL never tries to reinstall it.
-Andy
On Wed, Feb 14, 2018 at 11:56 AM, Jonathan Casco
<jca...@fiu.edu<mailto:jca...@fiu.edu>> wrote:
Hello,
I am reading through the documentation for xCAT in VCL but was a little
confused on the purpose xCAT has here.
Is xCAT being used for provisioning VM hosts like VMware to then get managed by
a separate VCL server or does VCL use xCAT to create servers that would be used
for VCL guests?
Thank you,
-----------------------
Jonathan Casco
HPC System Administrator
Florida International University
--
Andy Kurth
Research Storage Specialist
NC State University
Office of Information Technology
P: 919-513-4090
311A
Hillsborough<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D311A-2BHillsborough-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=0Mz5KaUjFlgb7ODSsLhATg&m=fi8Y17kZ-dX6IpkeLBgX96yFIr0tuKRhqP50_x38gGg&s=Urjk8BrLFibfwZwVAez_eQKEeW5_LVR81mfk09lKllo&e=>
Building
Campus Box 7109
Raleigh, NC 27695
Error! Filename not specified.
--
Andy Kurth
Research Storage Specialist
NC State University
Office of Information Technology
P: 919-513-4090
311A Hillsborough Building
Campus Box 7109
Raleigh, NC 27695
[Image removed by sender.]
