My solution to this is to have the below defined in an XCAT syncfiles config:
/var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs
/var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/
And then in a postscript this happens on the node being (re)installed:
# Only reconfigure if the host's address is present in the mmsdrfs file
# which is synced from the mgt node at install time
if for i in $(hostname --all-ip-addresses); do
grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break
done
then
/usr/lpp/mmfs/bin/mmsdrrestore
else
rm /var/mmfs/gen/mmsdrfs
fi
-Tyler Trafford
________________________________________
From: [email protected]
<[email protected]> on behalf of Peter Childs
<[email protected]>
Sent: Monday, February 1, 2021 10:08 AM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image.
We used to run
mmsdrestore -p manager -R /usr/bin/scp
in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however
we disliked needing to put the private key for the whole cluster on every host,
We now use
mmsdrestore -N nodename
post-install from a management node to re-add the node to the cluster, so we
could stop xcat from distributing the private key for security reasons.
Ideally we would have like the postscript call a manual call back to do this
but have not as yet worked out how best to do this in xcat, so currently its a
manual task which is fine when our nodes are stateless, but is not possible
when your nodes are stateless.
My understanding is that xcat should have a hook to do this like the
pre-scripts to run one at the end but I'm yet to find it.
Peter Childs
________________________________________
From: [email protected]
<[email protected]> on behalf of Ruffner, Scott (jpr9c)
<[email protected]>
Sent: Friday, January 29, 2021 8:04 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image.
Thanks David! Slick solution.
--
Scott Ruffner
Senior HPC Engineer
UVa Research Computing
(434)924-6778(o)
(434)295-0250(h)
[email protected]
From: <[email protected]> on behalf of
"[email protected]" <[email protected]>
Reply-To: gpfsug main discussion list <[email protected]>
Date: Friday, January 29, 2021 at 2:52 PM
To: gpfsug main discussion list <[email protected]>
Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image.
We use mmsdrrestore after the node boots. In our case these are diskless nodes
provisioned by xCAT. The post install script takes care of ensuring infiniband
is lit up, and does the mmsdrrestore followed by mmstartup.
-- ddj
Dave Johnson
On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) <[email protected]> wrote:
Hi everyone,
We want all of our compute nodes (bare metal) to directly participate in the
cluster as client nodes; of course, they are sharing a common root image.
Adding nodes via the regular mmaddnode (with the dsh operation to replicate
files to the clients) isn’t really viable, but if I short-circuit that, and
simply generate the /var/mmfs/gen files and then manually copy those and the
keyfiles to the shared root images, is that safe?
Am I going about this the entirely wrong way?
--
Scott Ruffner
Senior HPC Engineer
UVa Research Computing
(434)924-6778(o)
(434)295-0250(h)
[email protected]
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
