To address this, we are using the syncfiles mechanism to copy an rc.local
file into the compute node (we could probably also put it directly into the
image)
This rc.local contains statements to mount the /tmp volume. We originally
also used it to partition and format the physical hard disk, but that
proved too dangerous when somebody accidentally ran the script on the
management node and wiped out the partition table... You could probably do
something similar with a swap partition.
Here is the script we are using.
A few notes:
- a prerequisite is an entry that mounts /dev/sda1 as /localscratch in
fstab.
- moving the content from /tmp to /localscratch/tmp actually isn't working
flawlessly; it is just good enough for our purposes.
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
# Let's see if the local disk is already formatted and set up - if so,
# we won't redo it.
if [ -d /localscratch/tmp ]
then
echo "Localscratch disk is already formatted"
else
umount /dev/sda1
##################################################################
# DANGER DANGER DANGER! This code can, and will, blindly destroy
# partitions on whatever computer it is run. Partition recovery
# is not easy.
##################################################################
# Format the disk in the compute node. Single partition, mounted
# as localscratch.
#dd if=/dev/zero of=/dev/sda bs=1M count=100
#parted -s /dev/sda mklabel gpt
#parted -s -a optimal /dev/sda mkpart primary ext3 0% 100%
#mkfs -t ext3 /dev/sda1
mount -a
fi
# Create the /tmp and /var/tmp directories.
for i in /tmp /var/tmp /var/log
do
# Make sure mv includes .dotfiles
shopt -s dotglob
mkdir -p /localscratch$i
chmod 755 /localscratch
case "$i" in
/tmp)
# The first digit in the mode is the sticky bit.
chmod 1777 /localscratch$i
;;
/var/tmp)
# The first digit in the mode is the sticky bit.
chmod 1777 /localscratch$i
;;
*)
chmod 755 /localscratch$i
esac
if [ ! -h $i ]
then
if [ -n "$(ls -A $i)" ]
then
mv $i/* /localscratch$i
fi
# In theory, the directory should be empty because we moved everything
# out of the way. But that may have failed if the localscratch directory
# was already used.
rm -rf $i
ln -sf /localscratch$i $i
fi
shopt -u dotglob
done
mkdir -p /localscratch/ansys
chmod 777 /localscratch/ansys
On Mon, Nov 27, 2017 at 6:57 AM, Gilad Berman <gber...@lenovo.com> wrote:
> 1. I use local disk for scratch and swap. Somethings logs as well
> (in this case you can think of it as sort of statelite, but from xCAT
> perspective, it is still stateless).
>
> 2. I took only the part that not relate to statelite from the
> instructions – not working.
>
>
>
> [image:
> http://lenovocentral.lenovo.com/marketing/branding/email_signature/images/gradient.gif]
>
> *Gilad Berman*
> HPC Architect
> Lenovo EMEA
>
> [image: Phone]+972-52-2554262 <+972%2052-255-4262>
> [image: Email]gber...@lenovo.com <gber...@lenovo.com>
>
>
>
> Lenovo.com <http://www.lenovo.com/>
> Twitter <http://twitter.com/lenovo> | Facebook
> <http://www.facebook.com/lenovo> | Instagram
> <https://instagram.com/lenovo> | Blogs <http://blog.lenovo.com/> | Forums
> <http://forums.lenovo.com/>
>
> [image: DCG-Hardware]
>
>
>
>
>
> *From:* Russ Auld [mailto:russa...@comcast.net]
> *Sent:* Monday, November 27, 2017 4:50 PM
> *To:* xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
> *Subject:* Re: [xcat-user] Local scratch for stateless compute nodes
>
>
>
> If you're using netboot and local disk, then isn't that "statelite"?
>
> Do the satellite instructions not work?
>
>
>
> On Nov 27, 2017 9:26 AM, Gilad Berman <gber...@lenovo.com> wrote:
>
> All,
>
>
>
> I would like to join this question –
>
> Does even localdisk works with stateless? From the docs it seems that
> should be supported (because it is under stateless), however –
>
> - the instructions are taken from statelite and refer to
> statelite code (litefile)
>
> - The rc.localdisk code is under statelite
>
> - In the linuximage man – “Partitionfile - Only available for
> diskful osimages and statelite osimages(localdisk enabled)“
>
>
>
> A very quick trial on my statless nodes results in nothing J, it seems
> there is simply no reference to localdisk with stateless.
>
>
>
> So, can someone please help clarify it?
>
>
>
> ** as always, there is a chance I missed something very basic and it
> should be working J
>
>
>
> THX in advance!
>
>
>
> [image:
> http://lenovocentral.lenovo.com/marketing/branding/email_signature/images/gradient.gif]
>
> *Gilad Berman*
> HPC Architect
> Lenovo EMEA
>
> [image: Phone]+972-52-2554262 <+972%2052-255-4262>
> [image: Email]gber...@lenovo.com <gber...@lenovo.com>
>
>
>
> Lenovo.com <http://www.lenovo.com/>
> Twitter <http://twitter.com/lenovo> | Facebook
> <http://www.facebook.com/lenovo> | Instagram
> <https://instagram.com/lenovo> | Blogs <http://blog.lenovo.com/> | Forums
> <http://forums.lenovo.com/>
>
> [image: DCG-Hardware]
>
>
>
>
>
> *From:* Vinícius Ferrão [mailto:fer...@versatushpc.com.br
> <fer...@versatushpc.com.br>]
> *Sent:* Wednesday, November 22, 2017 4:09 AM
> *To:* xcat-user@lists.sourceforge.net
> *Subject:* [xcat-user] Local scratch for stateless compute nodes
>
>
>
> Hello,
>
>
>
> I would like to enable swap and local /tmp on my stateless nodes, but
> after following the documentation on the following link nothing appears to
> work:
>
> http://xcat-docs.readthedocs.io/en/stable/advanced/
> hierarchy/provision/diskless_sn.html
>
>
>
> I’m aware that the documentation is for service nodes and not for compute
> nodes, but I was thinking the procedure would be similar.
>
>
>
> At this point I’m with this settings on osimage:
>
> [root@headnode xcat]# lsdef -t osimage centos7.4-x86_64-netboot-compute
> exlist=/opt/xcat/share/xcat/netboot/centos/compute.centos7.exlist
> imagetype=linux
> osarch=x86_64
> osdistroname=centos7.4-x86_64
> osname=Linux
> osvers=centos7.4
> otherpkgdir=/install/post/otherpkgs/centos7.4/x86_64
>
> **** partitionfile=/install/custom/netboot/centos7.4-x86_
> 64-netboot-compute/partitionfile
>
> permission=755
> pkgdir=/install/centos7.4/x86_64
> pkglist=/opt/xcat/share/xcat/netboot/centos/compute.centos7.pkglist
> postinstall=/opt/xcat/share/xcat/netboot/centos/compute.
> centos7.postinstall
> profile=compute
> provmethod=netboot
> rootimgdir=/install/netboot/centos7.4/x86_64/compute
> synclists=/install/custom/netboot/compute.synclist
>
>
>
> And the content of the partition file is the following:
>
> cat /install/custom/netboot/centos7.4-x86_64-netboot-compute/partitionfile
> enable=yes
> enablepart=yes
>
> [disk]
> dev=/dev/sda
> clear=yes
> parts=10,90
>
> [swapspace]
> dev=/dev/sda1
>
> [localspace]
> dev=/dev/sda2
> fstype=xfs
>
>
>
> Finally the following commands were executed:
>
> chtab priority=7.1 policy.commands=getpartition policy.rule=allow
>
> chtab litefile.image=centos7.4-x86_64-netboot-compute
> litefile.file=/var/log/ litefile.options=localdisk
>
> chtab litefile.image=centos7.4-x86_64-netboot-compute litefile.file=/tmp/
> litefile.options=localdisk
>
>
>
> After a new genimage/packimage the local scratch does not appears to be
> made nor working.
>
>
>
> Someone knows what to do next?
>
>
>
> Thanks,
>
> V.
>
>
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
>
--
_______________________________________________________________________
Kevin Keane | Systems Architect | University of San Diego ITS |
kke...@sandiego.edu
Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 | 619.260.6859
<%28619%29%20260-2298>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
