Actually, I take it back, some of the nodes have the MN's IP in their
resolv.conf (which has no DNS server) after a kickstart install, which is
possibly why the localhost assignment happens since a lookup would fail. I
think only if I run a postscript that I specifically created to edit the
/etc/resolv.conf and point it to the SNs which run DNS servers does it
allow the node to resolve its name. I wish the kickstart would populate the
resolv.conf file with its service node IPs by default (and I thought it
used to??).
On Tue, Mar 8, 2016 at 4:26 PM, Josh Nielsen <[email protected]>
wrote:
> Here is what I see in /var/log/messages when remoteshell is run:
>
> Mar 7 14:28:41 xcat-serv1 node0087c xcat: remoteshell: setup
> /etc/ssh/sshd_config and ssh_config
> Mar 7 14:28:41 xcat-serv1 node0087c xcat: Install: setup root .ssh
> Mar 7 14:28:42 xcat-serv1 xcat-serv1 xCAT[16025]: xCAT: Allowing
> getcredentials ssh_dsa_hostkey from node0087c
> Mar 7 14:28:42 xcat-serv1 node0087c xCAT: remoteshell: getting
> ssh_host_dsa_key
> Mar 7 14:28:42 xcat-serv1 xcat-serv1 xCAT[16027]: xCAT: Allowing
> getcredentials ssh_rsa_hostkey from node0087c
> Mar 7 14:28:42 xcat-serv1 node0087c xCAT: ssh_rsa_hostkey
> Mar 7 14:28:42 xcat-serv1 node0087c xCAT: start up sshd
>
> I see new timestamps on authorized_keys and copy.sh when it is run, so it
> is actually doing something.
>
> Is there a substantial difference between remoteshell and updatenode -k?
> Why does updatenode -k successfully copy the id_rsa key to the node if I
> type in the password?
>
> As for the hostname in /etc/sysconfig/network, actually if I run just the
> kickstart and remove ifcfg-eth from the list of postscripts (in the node
> definition's postscripts= field) to execute automatically it ends up
> looking like this:
> # cat /etc/sysconfig/network
> NETWORKING=yes
> HOSTNAME=3(NXDOMAIN)
>
> But when I manually run updatenode node0087c -P ifcfg-eth it works
> correctly. (Also it changes the /etc/sysconfig/network-scripts/ifcfg-eth0
> file's BOOTPROTO parameter from dhcp to static and sets the IPADDR, as it
> should).
> # cat /etc/sysconfig/network
> NETWORKING=yes
> HOSTNAME=node0087c.morgan.haib.org
>
> Only if I keep the ifcfg-eth postscript in the node definition (listed in
> postscripts= if you 'lsdef') to be automatically executed does
> that /etc/sysconfig/network file say 'localhost', which seems to indicate
> an order of execution problem to me (even though I made sure ifcfg-eth was
> listed last). And, to answer your question, an nslookup node0087c before
> and after both return the correct IP from either of the SNs' slave DNS
> servers.
>
> They are possibly two unrelated issues, but I'm close to just upgrading
> xCAT and seeing if I have any better luck if I can't figure out some
> obvious problem soon. For kicks I'll explicitly set sshbetweennodes in the
> site table and rerun the remoteshell postscript.
>
> Regards,
> Josh
>
> On Tue, Mar 8, 2016 at 3:33 PM, Casandra H Qiu <[email protected]> wrote:
>
>> mmm, I don't have system with xCAT 2.8.3. but I think sshbetweenodes
>> attribute is available for while. if it is not defined in the site table,
>> the default should be set up passwordless between nodes.
>> "nslookup nodename" still works after you update the hostname, right?
>> can u able to find any error message from logs? maybe in the
>> /var/log/message.
>>
>> Thanks,
>> Casandra
>> ...................................................................
>> Casandra Hong Qiu
>> Phone: (845) 433-9291, t/l 293-9291
>> Office: B/002, Floor 3, Z13
>> [email protected]
>>
>>
>>
>> [image: Inactive hide details for Josh Nielsen ---03/08/2016 03:58:33
>> PM---Thanks for the response Casandra. I should firstly note that]Josh
>> Nielsen ---03/08/2016 03:58:33 PM---Thanks for the response Casandra. I
>> should firstly note that I have xCAT 2.8.3. I know I need to upg
>>
>> From: Josh Nielsen <[email protected]>
>> To: xCAT Users Mailing list <[email protected]>
>> Date: 03/08/2016 03:58 PM
>> Subject: Re: [xcat-user] Updatenode -k won't create id_rsa key without
>> prompting for password
>> ------------------------------
>>
>>
>>
>>
>> Thanks for the response Casandra. I should firstly note that I have xCAT
>> 2.8.3. I know I need to upgrade, but not only has this worked in the past
>> but I also successfully deployed 50+ nodes back in November with the exact
>> same xCAT version I have now and using the same osimage for Centos 6.5,
>> same kickstart, same defined postscripts, etc. So something else has
>> changed, perhaps in our environment?
>>
>> That being said, I did not see sshbetweennodes specified at all in the
>> site table. The following are the only two references to ssh in the table:
>>
>> #tabdump site | grep -i ssh
>> "maxssh","8",,
>> "rsh","/usr/bin/ssh",,
>>
>> Is 'sshbetweennodes' only a feature of versions newer than 2.8.X, or has
>> it been around a while?
>>
>> Lastly, you said that remoteshell copies over id_rsa.pub (regardless - in
>> either scenario), and I have seen that before as well, but actually I am
>> not seeing any id_rsa* keys (public or private) copied to the node at all.
>> And even an updatenode -k is only producing the id_rsa (if I manually type
>> the password) but not the .pub, which is also odd. But authorized_keys is
>> populated with the rsa public key signature. Something else must be going
>> on.
>>
>> P.S. The only other issue I'm still dealing with, which may irrelevant
>> for this issue, is a hostname problem to where if I run the ifcfg-eth
>> postscript it updates the hostname in /etc/sysconfig/network from the
>> correct node name to "localhost". My forward and reverse lookup entries in
>> DNS are present, and the hostname is set correctly by the kickstart before
>> ifcfg-eth is run, and remains there if it is not run, which I presume it
>> gets from either the node definition in dhcpd.leases which is created with
>> 'makedhcp' and/or the DNS entries for the host's IP. On the off chance that
>> key copying could be tied to name resolution inconsistencies I thought I
>> might mention that as well.
>>
>> Thanks,
>> Josh
>>
>> On Tue, Mar 8, 2016 at 1:20 PM, Casandra H Qiu <*[email protected]*
>> <[email protected]>> wrote:
>>
>> can u check the site table if sshbetweennodes is set up? The default
>> for sshbetweennodes is ALLGROUPS, and will enable passwordless between
>> nodes. this attribute will be ignored if zone table is set up, so please
>> check zone table also.
>>
>> if it enables, the remoteshell postscript will copy id_rsa and
>> id_rsa.pub over to compute node, otherwise, it only copies id_rsa.pub.
>>
>> from source code, updatenode -k is always required password.
>>
>>
>> Thanks,
>> Casandra
>> ...................................................................
>> Casandra Hong Qiu
>> Phone: *(845) 433-9291* <%28845%29%20433-9291>, t/l 293-9291
>> Office: B/002, Floor 3, Z13
>> *[email protected]* <[email protected]>
>>
>>
>>
>> [image: Inactive hide details for Josh Nielsen ---03/08/2016 12:51:58
>> PM---Yes, I just verified. It is present, but that alone is not s]Josh
>> Nielsen ---03/08/2016 12:51:58 PM---Yes, I just verified. It is present,
>> but that alone is not sufficient for that node to be able to SS
>>
>> From: Josh Nielsen <*[email protected]*
>> <[email protected]>>
>> To: xCAT Users Mailing list <*[email protected]*
>> <[email protected]>>
>> Date: 03/08/2016 12:51 PM
>> Subject: Re: [xcat-user] Updatenode -k won't create id_rsa key
>> without prompting for password
>> ------------------------------
>>
>>
>>
>> Yes, I just verified. It is present, but that alone is not sufficient
>> for that node to be able to SSH to other nodes itself. It allows other
>> nodes which have the correct private key to SSH to it, but not the other
>> way around.
>>
>> For example, on one compute node I'm having trouble with /root/.ssh
>> has these three files:
>>
>> -rw-------. 1 root root 408 Mar 7 14:28 authorized_keys
>> -rw-------. 1 root root 411 Mar 7 14:28 copy.sh
>> -rw------- 1 root root 402 Mar 3 16:20 known_hosts
>>
>> And authorized_keys has the correct ssh-rsa public key entry, but I
>> cannot go from this node to any other node in my cluster via passwordless
>> ssh. But as soon as I run updatenode -k, and type in the password that it
>> prompts for to complete the command, the id_rsa key is added as the fourth
>> file to the /root/.ssh directory, and then after that I can ssh to other
>> nodes from it without supplying a password. That is the issue.
>>
>> In the past simply running the remoteshell postscript (or so I
>> assumed) was sufficient for adding the id_rsa file, and it was all
>> automated from a fresh deploy by specifying remoteshell as one of the
>> default postscripts to run. But now it doesn't look like remoteshell is
>> placing the id_rsa file on the node (unless some other script or command
>> is
>> responsible for that), but remoteshell looks like it creates everything
>> else in /root/.ssh/ (and /etc/ssh/).
>>
>> Is remoteshell the correct postscript for that, or was the id_rsa key
>> most likely being pushed to the nodes some other way (like by some code
>> that called updatenode -k upon initial deployment)? Either way, all I can
>> say for sure is that id_rsa used to appear in /root/.ssh on the compute
>> node automatically and now it does not.
>>
>> Regards,
>> Josh
>>
>> On Tue, Mar 8, 2016 at 4:19 AM, Xiao Peng Wang <*[email protected]*
>> <[email protected]>> wrote:
>> To enable the login without password, the rsa public key should be
>> copied to /root/.ssh/authorized_keys in the compute node. Could you
>> check
>> whether the key has been added in to
>> /root/.ssh/authorized_keys?
>>
>>
>>
>> Thanks
>> Best Regards
>>
>>
>> ----------------------------------------------------------------------
>> Wang Xiaopeng (王晓朋)
>> IBM China System Technology Laboratory
>> Tel: 86-10-82453455
>> Email: *[email protected]* <[email protected]>
>> Address: 28,ZhongGuanCun Software Park,No.8 Dong Bei Wang West
>> Road, Haidian District Beijing P.R.China 100193
>>
>>
>> ----- Original message -----
>> From: Josh Nielsen <*[email protected]*
>> <[email protected]>>
>> To: xCAT Users Mailing list <*[email protected]*
>> <[email protected]>>
>> Cc:
>> Subject: Re: [xcat-user] Updatenode -k won't create id_rsa key
>> without prompting for password
>> Date: Tue, Mar 8, 2016 5:26 AM
>>
>> Also if remoteshell is invoked directly as a
>> postscript ('updatenode node0086c -V -P remoteshell') it produces
>> the same
>> result, but does not prompt for a password (like invoking xdsh -K
>> directly
>> doesn't), and copies everything over except id_rsa. So actually the
>> prompting for a password is specific to updatenode -k, not xdsh -K
>> or the
>> remoteshell postscript (which run that). So I'm not sure if that is
>> relevant to the underlying problem or not, but if I do invoke
>> updatenode -k
>> and supply it the password it copies the id_rsa to the node.
>>
>> On Mon, Mar 7, 2016 at 2:12 PM, Josh Nielsen <
>> *[email protected]* <[email protected]>> wrote:
>> Hello,
>>
>> When we freshly deploy a node from the kickstart and run our
>> postscripts we noticed that for some reason the /root/.ssh/id_rsa
>> file
>> which allows passwordless login from that node to other nodes is
>> missing,
>> though this was not the case just a few months ago. When I try to
>> generate
>> the key manually it prompts for a password, after which it will
>> copy/create
>> that file successfully (see below), but there are a few odd things
>> connected to this.
>>
>> The error is:
>> updatenode node0087c -k
>> Enter the password for the userid: root on the node where the
>> ssh keys will be updated:
>>
>> The first oddity is that even after supplying the password once
>> for a particular node it will prompt for the password every time if
>> I run
>> it again, as well as the related problem that this never used to
>> happen
>> before and the key used to be created without issue or prompting
>> for a
>> password. The 'passwd' xCAT table has the password for root (if
>> that is
>> where it looks for this command).
>>
>> Secondly I have done several manual debugging steps (and poking
>> around source code to see what is happening) and I have run the
>> actual xdsh
>> command that is called, shown from the -V verbose output (which it
>> prints
>> two of, the first apparently to prep the SNs and run the
>> 'remoteshell'
>> postscript on them, and the second to actually do the same to the
>> node
>> specified).
>>
>> xdsh sn1,sn2 --nodestatus -s -v -e
>> /install/postscripts/xcatdsklspost 5 -m [MN_IP]
>> 'remoteshell,servicenode'
>> --tftp /tftpboot --installdir /install --nfsv4 no -c -V
>>
>> xdsh node0086c --nodestatus -s -v -e
>> /install/postscripts/xcatdsklspost 5 -m [SN1_IP] 'remoteshell'
>> --tftp
>> /tftpboot --installdir /install --nfsv4 no -c -V
>>
>> This did not reveal anything useful, except that when invoked
>> directly like this no password is prompted for and it runs, but
>> still
>> leaves out the id_rsa file. I followed also the suggestion by Wang
>> Xaiopeng
>> in this thread (*http://tinyurl.com/jz2jzmb*
>> <http://tinyurl.com/jz2jzmb>*)* to test the getcredentials call
>> with:
>>
>> 1. Enable mini server
>> /xcatpost/allowcred.awk &
>>
>> 2.Try to get rsa hostkey
>> USEOPENSSLFORXCAT=yes XCATSERVER=<MNIP>:3001
>> /xcatpost/getcredentials.awk ssh_rsa_hostkey
>>
>> This returned ssh_rsa_hostkey sucessfully. When remoteshell is
>> run (whether with updatenode -k or xdsh -K) it actually does copy
>> over the
>> key files into /etc/ssh/ and it copies known_hosts, copy.sh, and
>> authorized_keys into /root/.ssh on the compute node but omits
>> id_rsa. What
>> could be going wrong here?
>>
>> Regards,
>> Josh Nielsen
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> *http://makebettercode.com/inteldaal-eval*
>> <http://makebettercode.com/inteldaal-eval>
>> _______________________________________________
>> xCAT-user mailing list
>> *[email protected]* <[email protected]>
>> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
>> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> *http://makebettercode.com/inteldaal-eval*
>> <http://makebettercode.com/inteldaal-eval>
>> _______________________________________________
>> xCAT-user mailing list
>> *[email protected]* <[email protected]>
>> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
>> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> *http://makebettercode.com/inteldaal-eval*
>> <http://makebettercode.com/inteldaal-eval>
>> _______________________________________________
>> xCAT-user mailing list
>> *[email protected]* <[email protected]>
>> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
>> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> *http://makebettercode.com/inteldaal-eval*
>> <http://makebettercode.com/inteldaal-eval>
>> _______________________________________________
>> xCAT-user mailing list
>> *[email protected]* <[email protected]>
>> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
>> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>>
>>
>> ------------------------------------------------------------------------------
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> http://makebettercode.com/inteldaal-eval
>> _______________________________________________
>> xCAT-user mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> http://makebettercode.com/inteldaal-eval
>> _______________________________________________
>> xCAT-user mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>
>>
>
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
