Hiļ¼Thomas
Really long mail. ^_^
Your scenario is normal in a HPC cluster.
eth0 ==> carry both BMC and host data means it running in SHARED mode.
So, I'd like to make sure that eth0 and eth2 are in the same subnet.
Then, my explanation are blue as below.
But, I strongly suggest you to update to latest xCAT 2.14.5+patch https://github.com/xcat2/xcat-core/issues/5905#issuecomment-448165125 which have enhanced a lot for hardware discovery progress.
So, basically, I don't understand the conceptual difference between 1)
and 4) and thus why in 1) nodeA gets discovered without the need to run
nodediscoverstart (granted it has switch and swithport attributes but
not refering the 1Gb/s switch)...
Er Tao: for 1, since there is node: nodeA defined in xCAT database, and the mtm and serial have been updated by switch based node discovery, then discovery request can be matched with MTMS based discovery. No rmdef nodeA
and 4) and thus why in 1) nodeA gets discovered without the need to run
nodediscoverstart (granted it has switch and swithport attributes but
not refering the 1Gb/s switch)...
Er Tao: for 1, since there is node: nodeA defined in xCAT database, and the mtm and serial have been updated by switch based node discovery, then discovery request can be matched with MTMS based discovery. No rmdef nodeA
for 4, when a node need to be matched with sequentially, xCAT will find free nodes specified by nodediscoverstart noderange=foobar
Again, I'd like to understand the above in order to be sure that the
node won't get provisionned through the eth0/1Gb/s link if PXE order
gets mixed up...Or even to make sure (worst case scenario if many nodes
would have PXE order mixed up) that nodes hostnames won't be mixed up
(in a serial discovery like scenario).
Er Tao: If you update xCAT newer that 2.12.4, there is an option `noboot` for site.dhcpinterfaces to specify the NIC on MN that not reply DHCP configuration file for the DHCP request get from this NIC. So, it means, if MN eth0 connect to CN eth0/BMC, eth1 --> CN eth2.
Then, with the configuration below and `mknb <ppc64|x86_64>`, the CN won't get DHCP configuration from eth0.
chdef -t site dhcpinterfaces="eth0:noboot,eth1"Pls feel free to let me know if there is any more concerns.
Thx!
Best Regards,
-----------------------------------
Zhao Er Tao
IBM China System and Technology Laboratory, Beijing
Tel:(86-10)82450485
Email: erta...@cn.ibm.com
Address: 1/F, 28 Building,ZhongGuanCun Software Park,
No.8 DongBeiWang West Road, Haidian District,
Beijing, 100193, P.R.China
-----------------------------------
Zhao Er Tao
IBM China System and Technology Laboratory, Beijing
Tel:(86-10)82450485
Email: erta...@cn.ibm.com
Address: 1/F, 28 Building,ZhongGuanCun Software Park,
No.8 DongBeiWang West Road, Haidian District,
Beijing, 100193, P.R.China
----- Original message -----
From: Thomas HUMMEL <thomas.hum...@pasteur.fr>
To: xcat-user@lists.sourceforge.net
Cc:
Subject: [xcat-user] Discovery with and without nodediscoverstart
Date: Tue, Feb 12, 2019 10:19 PM
Hello,
I'm using xCAT 2.11 in production (I know it's an old release) to
provision HPC multi-interface compute nodes with stateless images.
Basically, the nodes have one 1Gb/s nic and one 10Gb/s nic, each one
connected to a different switch. Obvioulsy, I want them to be
provisioned through the 10Gb/s link.
I'm using switch-based discovery and, provided that BIOS PXE priorities
is correctly configured, everything works as expected.
But I was wondering what would happen if BIOS was wrongly configured to
PXE on the 1Gb/s nic first : would discover happen ? if yes, would it
result to ?
I first though it would be similar to a serial discovery situation but
it doesn't seem that simple, here's why :
The compute nodes are physical nodes equipped with
- 2 onboard 1Gb/s nic (eth0, eth1)
- 1 additionnal PCI-E 10Gb/s nic (eth2)
and set up like this :
- eth2 is connected to a 10Gb/s ports switch which is used by
switch-based discovery ('switch' and 'switchport' attributes point to
this switch and switch ports)
- eth0 is connected to a 1Gb/s ports switch and is used for ipmi traffic
from/to the BMC and could also carry the data traffic (legacy from the
time we didn't have a 10Gb/s switch nor nic")
- BIOS is set up to PXE boot first and PXE is set up to boot through
eth2 first.
Here's what I tested :
Starting conditions :
- Let's say nodeA was discovered and netbooted through the method above
(thus its mac in the mac table is eth2 (10Gb/s) address).
- let's say I haven't any undiscovered nor unprovisionned node
To see what would happen through eth0, I changed the eth2 facing switch
configuration to forbid traffic coming from this interface just to be
sure eth2 PXE could no work no matter what. So now the node does PXE on
eth0 (1Gb/s)
I know about the 3 types of automatic discovery as described in
documentation (MTMS, switch-based, serial) and here's what I tested :
1) nodeA :
- remove its mac address in the mac table
- remove all its leases in /var/lib/dhcpd/dhcpd.leases
- makedhcp -n
reboot nodeA
-> it is discovered and provisionned with its osimage but through the
eth0 interface. hostname is nodeA as expected
in the mac table we can see
"nodeA",,"<eth0 mac-address>|<eth2 mac-address>!*NOIP*",,
-> seems strange to me (see at the end of my tests description)
2) nodeA
- rmdef nodeA
- remove all its leases in /var/lib/dhcpd/dhcpd.leases
- makedhcp -n
- add a new node with simmilar properties (same chain, ...) :
nodeadd foobar groups=<same as nodeA>
tabedit hosts : insert a specific entry with an ip for foobar
tabedit ipmi : insert a specific entry with an bmc ip for foobar
makedns foobar
reboot nodeA
-> it gets stalled ("Network configuration complete, commencing transmit
of discovery packets") and PXE loops after getting an IP from the dhcpd
dynamic range
-> seems pretty normal to me since I consider myself to be in a serial
discovery like scenario then
3) same as 2) but with a nodediscoverstart noderange=foobar AFTER node
reboot : seems to loop in PXE/discovery (even if nodediscoverstop is run)
-> seems to me it might be normal as I maybe was supposed to run
nodediscoverstart before node boot.
4) same as 2) but with a nodediscoverstart noderange=foobar BEFORE node
reboot
-> foobar (same physical node as nodeA) gets discovered and
provisionned. hostname is foobar as expected
in the mac table we can see
"foobar",,"0c:c4:7a:4d:85:a8",,
-> seems pretty normal to me
[I don't quite understand what is happening in 3) but it a bit off topic]
So, basically, I don't understand the conceptual difference between 1)
and 4) and thus why in 1) nodeA gets discovered without the need to run
nodediscoverstart (granted it has switch and swithport attributes but
not refering the 1Gb/s switch)...
Again, I'd like to understand the above in order to be sure that the
node won't get provisionned through the eth0/1Gb/s link if PXE order
gets mixed up...Or even to make sure (worst case scenario if many nodes
would have PXE order mixed up) that nodes hostnames won't be mixed up
(in a serial discovery like scenario).
Thanks
--
Thomas HUMMEL
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
_______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
