I would suggest separating out your host name from you node names and
utilize the hostname fields and a simple postscript to change hostname
based upon it's ip lookup from dns simply done and keeps separation just
requires a little thought on layout
ie
hosts table
#node,ip,hostnames,otherinterfaces,comments,disable
"datanode","|\D+(\d+).*$|172.20.2.($1)|","|\D+(\d).*$|testcore($1)-a|",,,
"management","|\D+(\d+).*$|172.20.1.($1)|","",,,
"data10g","|\D+(\d+).*$|172.21.2.($1)|",|\D+(\d).*$|testcore($1)|,,,
"manage10g","|\D+(\d+).*$|172.21.1.($1)|",,,,
then
nodeadd n1-n7 groups=datanode,.....
which will give you defined nodes n1-n7 that aliases to testcore1-a through
testcore7-a
nodeadd testcore1-testcore7 groups=data10g
defined on your 10 gig keeeping from overlap of you 1Gb names
then
nodeset datanode osimage=rhel....
or
nodeset n1-n7 osimage=rhel...
which will always point to the 1gb interfaces.
the issue you run into is your depending on DNS lookup for similar names
for different nets the node names would be best served by separating them
out from each other.
then as a post script you could do
forr a RHEL based something like
IP_ADDR=`ifconfig ((10 gig interface)) |grep 'inet addr'|awk '{print $2}'|
cut -f2 -d:`
HostName=`host $IP_ADDR |awk '{print $4}'`
sed -i "s/HOSTNAME=.*/HOSTNAME=$HostName/" /etc/sysconfig/network
Message: 5
Date: Mon, 07 Oct 2013 18:00:08 -0500
From: Russell Jones <[email protected]>
Subject: Re: [xcat-user] xCAT not generating kcmdline + dhcpd lease
correctly
To: [email protected]
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252; format=flowed
Ah, I see that you are already setting it up with separate subdomains.
My apologies, missed that section when I was quickly skimming :-)
Seems like really the only issue is the search path on the headnode
seeing the 10 gig first before the 1 gig management network. Not sure if
there's a way around that without changing the search path on the
headnode and changing any jobs/applications you are running to look at
the appropriate subdomain as opposed to relying on search paths to get
the traffic on the fastest network.
On 10/7/2013 5:50 PM, Russell Jones wrote:
> > After setting that, everything looks fine, except for the lease
> written into /var/lib/dhcpd/dhcpd.leases (still has the 10.233 subnet,
> which is 10GbE).
>
>
> It sounds like you are not using the headnode for DNS management of the
> cluster address space, correct?
>
> It seems you are setting DNS up quite a bit differently than is usually
> done (in my experience). The short name with no extra dashes is usually
> used for the management network, then from there $HOSTNAME-eth1,
> $HOSTNAME-eth2, etc resolves to their appropriate interfaces.
>
> In addition, most sites I have done cluster management at use a separate
> sub/domain all together for the different networks. Example:
> node1.cluster = management, node1.10g.cluster = storage,
> node1.pub.cluster = public
>
> I'm not sure the best route to take for getting your IP addresses
> resolved in the appropriate order. Relying on the search path to ensure
> traffic gets where it needs to go seems like sort of a dangerous way of
> doing it.
>
>
>
>
>
> On 10/7/2013 4:55 PM, Steve Crusan wrote:
>> Russell,
>>
>> Thanks for the response. I think we've gotten closer. After
setting that, everything looks fine, except for the lease written
into /var/lib/dhcpd/dhcpd.leases (still has the 10.233 subnet, which is
10GbE).
>>
>> I'll pour through the dhcp.pm module to see if I find out
what's happening.
>>
>> xCAT aficionado's out there, this problem occurs when I run:
makedhcp $shortHostname. If you recall, I believe it's because xCAT does a
hostname lookup on the nodeName, which in our case resolves to 10.233.1.68,
due to how our DNS is setup (see first message).
>>
>> Any way to get around that?
>>
>> ~Steve
>>
>> On Oct 7, 2013, at 2:18 PM, Russell Jones wrote:
>>
>>> Set noderes.nfsserver to the correct IP address to use for your
>>> kickstart repo.
>>>
>>>
>>>
>>>
>>> On 10/7/2013 1:33 PM, Steve Crusan wrote:
>>>> Hi all,
>>>>
>>>> Our network setup is like so:
>>>> - 1Gb ethernet (deployment / xCAT / management network)
>>>> - 10Gb ethernet
>>>> - out-of-band (IMM/BMC)
>>>>
>>>> Right now, our DNS is handled by setting up search paths on
our xcat server (via /etc/resolv.conf), with the *fastest* paths being
resolved first. So, 10Gb is resolved before 1Gb, like such:
>>>>
>>>> $ cat /etc/resolv.conf
>>>> nameserver 10.230.16.5
>>>> nameserver 10.230.16.6
>>>> search fst.domain.com mcp.domain.com lom.domain.com i2b.domain.com
domain.com
>>>> $ host testcoreb
>>>> testcoreb.fst.domain.com has address 10.233.1.68
>>>> $ host testcoreb.mcp
>>>> testcoreb.mcp.domain.com has address 10.230.1.68
>>>>
>>>> 10.233 is 10Gb ethernet, and 10.230 is 1Gb ethernet. We DO NOT want to
provision/manage (psh, etc) over 10Gb, but rather the 1G ethernet. We don't
use slashes in our short hostnames (hostname-mgmt), which I think is
causing a problem. Some hosts that do not have a 10GbE provision fine,
because the short name lookup defaults to that node's 10.230 network, which
is correct:
>>>> $ host fsm07
>>>> fsm07.mcp.domain.com has address 10.230.1.87
>>>>
>>>> We only have dhcpd listening on 1Gb ethernet as well:
>>>> $ tabdump site | grep dhcp
>>>> "dhcpinterfaces","eth0",,
>>>>
>>>> When I do a nodeset testcoreb osimage=imageName, the 10.233 interface
is used, which is wrong:
>>>> kcmdline=quiet repo=http://10.233.1.5:80/install/rhels6.4/x86_64
ks=http://10.233.1.5:80/install/autoinst/testcoreb.mcp
ksdevice=40:f2:e9:0c:ef:04 cmdline console=tty0 console=ttyS19200,hard
>>>>
>>>>> From dhcpd.leases:
>>>> host testcoreb {
>>>> dynamic;
>>>> hardware ethernet 40:f2:e9:0c:ef:04;
>>>> fixed-address 10.233.1.68;
>>>> supersede server.ddns-hostname = "testcoreb";
>>>> <SNIPPED>
>>>>
>>>> Is there a way to explicitly tell xCAT *not* to do a DNS lookup for
the shortname, and determine which interface to set for the tftp server?
I've tried the following in the noders table, without any success:
>>>>
>>>> $ tabdump noderes | grep testcoreb
>>>>
"testcoreb",,"pxe","10.230.1.5",,"10.230.1.5","10.230.1.5",,"mac","mac",,,,,,,,,
>>>>
>>>> I've tried manually setting all of this in the bootparam table, etc,
without any success.
>>>>
>>>> So, why long-winded question is, am I just missing setting a key
somewhere in one of the 40+ xCAT tables to override the short name lookups,
and explicitly use the 1Gb ethernet network? I haven't even defined the
10Gb ethernet network in the xCAT networks table either?
>>>>
>>>> Thanks,
>>>>
>>>> ~Steve
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
------------------------------------------------------------------------------
>>>> October Webinars: Code for Performance
>>>> Free Intel webinars can help you accelerate application performance.
>>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the
most from
>>>> the latest Intel processors and coprocessors. See abstracts and
register >
>>>>
http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> xCAT-user mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>>>
>>>
------------------------------------------------------------------------------
>>> October Webinars: Code for Performance
>>> Free Intel webinars can help you accelerate application performance.
>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the
most from
>>> the latest Intel processors and coprocessors. See abstracts and
register >
>>>
http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> xCAT-user mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>> --------------------------------------
>> Steve Crusan
>> HPC System Administrator - Filesystems
>> Leadership Computing Facility
>> Argonne National Laboratory
>>
>>
>>
>>
>>
>>
------------------------------------------------------------------------------
>> October Webinars: Code for Performance
>> Free Intel webinars can help you accelerate application performance.
>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
from
>> the latest Intel processors and coprocessors. See abstracts and register
>
>>
http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
>> _______________________________________________
>> xCAT-user mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>
>
>
------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
from
> the latest Intel processors and coprocessors. See abstracts and register
>
>
http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
> _______________________________________________
> xCAT-user mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
------------------------------
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
------------------------------
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
End of xCAT-user Digest, Vol 50, Issue 11
*****************************************
Thanks, Chris Eckhoff
IBM STG Lab Services - HPC Linux Clusters
[email protected]
RHCE, Novell CLP, A+, Server +
Phone# (720) 396-2734 UM Tie line Number: 938-2734
ibm.com/systems/services/labservices------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
