We’ve used similar scripts in the past without any checks to prevent unintended
disasters. It would be pretty easy to use an if or case statement to ensure
anything destructive only happens on the right systems.
Regards,
Christian Caruthers
Lenovo Professional Services
Mobile: 757-289-9872
From: Vinícius Ferrão [mailto:fer...@versatushpc.com.br]
Sent: Monday, November 27, 2017 2:17 PM
To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
Subject: Re: [xcat-user] Local scratch for stateless compute nodes
Very good info indeed. I will be looking on the script, Kevin.
It would be sufficient for a while, but about the feature, it would be nice to
be fixed/documented/explained how to use it, because it’s a pretty common use
case of stateless nodes.
Perhaps someone on the dev team can look at this? Should we open a ticket on
the issue tracker?
V.
On 27 Nov 2017, at 17:02, Gilad Berman
<gber...@lenovo.com<mailto:gber...@lenovo.com>> wrote:
THX Kevin for the tip!!
We actually used similar method, but the first post on the thread reminded me
of the localdisk feature and I thought it can be very nice to use it, if
working.
<image001.jpg>
Gilad Berman
HPC Architect
Lenovo EMEA
<image002.png>+972-52-2554262
<image003.png>gber...@lenovo.com<mailto:gber...@lenovo.com>
Lenovo.com <http://www.lenovo.com/>
Twitter<http://twitter.com/lenovo> | Facebook<http://www.facebook.com/lenovo> |
Instagram<https://instagram.com/lenovo> | Blogs<http://blog.lenovo.com/> |
Forums<http://forums.lenovo.com/>
<image004.jpg>
From: Kevin Keane [mailto:kke...@sandiego.edu]
Sent: Monday, November 27, 2017 7:22 PM
To: xCAT Users Mailing list
<xcat-user@lists.sourceforge.net<mailto:xcat-user@lists.sourceforge.net>>
Subject: Re: [xcat-user] Local scratch for stateless compute nodes
To address this, we are using the syncfiles mechanism to copy an rc.local file
into the compute node (we could probably also put it directly into the image)
This rc.local contains statements to mount the /tmp volume. We originally also
used it to partition and format the physical hard disk, but that proved too
dangerous when somebody accidentally ran the script on the management node and
wiped out the partition table... You could probably do something similar with a
swap partition.
Here is the script we are using.
A few notes:
- a prerequisite is an entry that mounts /dev/sda1 as /localscratch in fstab.
- moving the content from /tmp to /localscratch/tmp actually isn't working
flawlessly; it is just good enough for our purposes.
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
# Let's see if the local disk is already formatted and set up - if so,
# we won't redo it.
if [ -d /localscratch/tmp ]
then
echo "Localscratch disk is already formatted"
else
umount /dev/sda1
##################################################################
# DANGER DANGER DANGER! This code can, and will, blindly destroy
# partitions on whatever computer it is run. Partition recovery
# is not easy.
##################################################################
# Format the disk in the compute node. Single partition, mounted
# as localscratch.
#dd if=/dev/zero of=/dev/sda bs=1M count=100
#parted -s /dev/sda mklabel gpt
#parted -s -a optimal /dev/sda mkpart primary ext3 0% 100%
#mkfs -t ext3 /dev/sda1
mount -a
fi
# Create the /tmp and /var/tmp directories.
for i in /tmp /var/tmp /var/log
do
# Make sure mv includes .dotfiles
shopt -s dotglob
mkdir -p /localscratch$i
chmod 755 /localscratch
case "$i" in
/tmp)
# The first digit in the mode is the sticky bit.
chmod 1777 /localscratch$i
;;
/var/tmp)
# The first digit in the mode is the sticky bit.
chmod 1777 /localscratch$i
;;
*)
chmod 755 /localscratch$i
esac
if [ ! -h $i ]
then
if [ -n "$(ls -A $i)" ]
then
mv $i/* /localscratch$i
fi
# In theory, the directory should be empty because we moved everything
# out of the way. But that may have failed if the localscratch directory
# was already used.
rm -rf $i
ln -sf /localscratch$i $i
fi
shopt -u dotglob
done
mkdir -p /localscratch/ansys
chmod 777 /localscratch/ansys
On Mon, Nov 27, 2017 at 6:57 AM, Gilad Berman
<gber...@lenovo.com<mailto:gber...@lenovo.com>> wrote:
1. I use local disk for scratch and swap. Somethings logs as well (in this
case you can think of it as sort of statelite, but from xCAT perspective, it is
still stateless).
2. I took only the part that not relate to statelite from the instructions
– not working.
<image009.jpg>
Gilad Berman
HPC Architect
Lenovo EMEA
<image010.png>+972-52-2554262<tel:+972%2052-255-4262>
<image011.png>gber...@lenovo.com<mailto:gber...@lenovo.com>
Lenovo.com <http://www.lenovo.com/>
Twitter<http://twitter.com/lenovo> | Facebook<http://www.facebook.com/lenovo> |
Instagram<https://instagram.com/lenovo> | Blogs<http://blog.lenovo.com/> |
Forums<http://forums.lenovo.com/>
<image012.jpg>
From: Russ Auld [mailto:russa...@comcast.net<mailto:russa...@comcast.net>]
Sent: Monday, November 27, 2017 4:50 PM
To: xCAT Users Mailing list
<xcat-user@lists.sourceforge.net<mailto:xcat-user@lists.sourceforge.net>>
Subject: Re: [xcat-user] Local scratch for stateless compute nodes
If you're using netboot and local disk, then isn't that "statelite"?
Do the satellite instructions not work?
On Nov 27, 2017 9:26 AM, Gilad Berman
<gber...@lenovo.com<mailto:gber...@lenovo.com>> wrote:
All,
I would like to join this question –
Does even localdisk works with stateless? From the docs it seems that should be
supported (because it is under stateless), however –
- the instructions are taken from statelite and refer to statelite code
(litefile)
- The rc.localdisk code is under statelite
- In the linuximage man – “Partitionfile - Only available for diskful
osimages and statelite osimages(localdisk enabled)“
A very quick trial on my statless nodes results in nothing ☺, it seems there is
simply no reference to localdisk with stateless.
So, can someone please help clarify it?
** as always, there is a chance I missed something very basic and it should be
working ☺
THX in advance!
Gilad Berman
HPC Architect
Lenovo EMEA
+972-52-2554262<tel:+972%2052-255-4262>
gber...@lenovo.com<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
Lenovo.com
Twitter | Facebook | Instagram | Blogs | Forums<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
From: Vinícius Ferrão [mailto:fer...@versatushpc.com.br]
Sent: Wednesday, November 22, 2017 4:09 AM
To: xcat-user@lists.sourceforge.net
Subject: [xcat-user] Local scratch for stateless compute
nodes<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
Hello,<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
I would like to enable swap and local /tmp on my stateless nodes, but after
following the documentation on the following link nothing appears to
work:<mailto:gber...@lenovo.com>
http://xcat-docs.readthedocs.io/en/stable/advanced/hierarchy/provision/diskless_sn.html<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
I’m aware that the documentation is for service nodes and not for compute
nodes, but I was thinking the procedure would be
similar.<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
At this point I’m with this settings on osimage:<mailto:gber...@lenovo.com>
[root@headnode xcat]# lsdef -t osimage centos7.4-x86_64-netboot-compute
exlist=/opt/xcat/share/xcat/netboot/centos/compute.centos7.exlist
imagetype=linux
osarch=x86_64
osdistroname=centos7.4-x86_64
osname=Linux
osvers=centos7.4
otherpkgdir=/install/post/otherpkgs/centos7.4/x86_64<mailto:gber...@lenovo.com>
****
partitionfile=/install/custom/netboot/centos7.4-x86_64-netboot-compute/partitionfile<mailto:gber...@lenovo.com>
permission=755
pkgdir=/install/centos7.4/x86_64
pkglist=/opt/xcat/share/xcat/netboot/centos/compute.centos7.pkglist
postinstall=/opt/xcat/share/xcat/netboot/centos/compute.centos7.postinstall
profile=compute
provmethod=netboot
rootimgdir=/install/netboot/centos7.4/x86_64/compute
synclists=/install/custom/netboot/compute.synclist<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
And the content of the partition file is the
following:<mailto:gber...@lenovo.com>
cat /install/custom/netboot/centos7.4-x86_64-netboot-compute/partitionfile
enable=yes
enablepart=yes
[disk]
dev=/dev/sda
clear=yes
parts=10,90
[swapspace]
dev=/dev/sda1
[localspace]
dev=/dev/sda2
fstype=xfs<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
Finally the following commands were executed:<mailto:gber...@lenovo.com>
chtab priority=7.1 policy.commands=getpartition
policy.rule=allow<mailto:gber...@lenovo.com>
chtab litefile.image=centos7.4-x86_64-netboot-compute litefile.file=/var/log/
litefile.options=localdisk<mailto:gber...@lenovo.com>
chtab litefile.image=centos7.4-x86_64-netboot-compute litefile.file=/tmp/
litefile.options=localdisk<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
After a new genimage/packimage the local scratch does not appears to be made
nor working.<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
Someone knows what to do next?<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
Thanks,<mailto:gber...@lenovo.com>
V.<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
<mailto:gber...@lenovo.com>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user<mailto:gber...@lenovo.com>
-- <mailto:gber...@lenovo.com>
_______________________________________________________________________
Kevin Keane | Systems Architect | University of San Diego ITS |
kke...@sandiego.edu
Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 |
619.260.6859<mailto:gber...@lenovo.com>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org<http://slashdot.org/>!
http://sdm.link/slashdot_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net<mailto:xCAT-user@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
