if nslookup works fine, the name resolution should not have problem. but
you can run "makedns -n" , "makedhcp -n" and "makedhcp -a" again.
the remoteshell postscript is the file copies SSH Key over to compute
nodes, should be default postscripts which defined in your node
definitions,
you can add "set -x" to this file, the log information will be on the
node /var/log/xcat/xcat.log
u can show me the site table, network table and lsdef for one of nodes if
nothing works? :(
Thanks,
Casandra
...................................................................
Casandra Hong Qiu
Phone: (845) 433-9291, t/l 293-9291
Office: B/002, Floor 3, Z13
[email protected]
From: Josh Nielsen <[email protected]>
To: xCAT Users Mailing list <[email protected]>
Date: 03/09/2016 11:16 AM
Subject: Re: [xcat-user] Updatenode -k won't create id_rsa key without
prompting for password
I should also mention that for kicks last week we tried deploying one VM
that had been defined in the hpvmgroupA group 4 months ago with the rest of
them, but had not been deployed and had been delayed until now, and it too
is experiencing the same problems with the keys as well as our (unrelated?)
hostname issues, so I'm going for something that changed in our
environment, be it networking related, DNS, perhaps permissions, or any
host of other variables that can change on systems involved with the xCAT
deploy process. I just wish there were a meaningful error or way to
determine the root issue more easily.
On Wed, Mar 9, 2016 at 10:01 AM, Josh Nielsen <[email protected]>
wrote:
Yes. Sorry that I did not mention it before, but they are stateful nodes.
I have never used stateless nodes and have nothing configured for
stateless. I'm just puzzled what could have changed in four months since
I last I deployed 50+ VMs with the same osimage template and postscripts
they used at that time. I'm deploying Centos 6.5 to a series of VMs on
ESXi (standalone - not vSphere managed) on an HP CX7000 Blade chassis.
I'm PXE booting as the method of delivering the kickstart, and I'm only
having problems post-kickstart (or so it seems...). And the deploy is
completely abstracted to the VM guest operating system level, so nothing
specific to HP.
I'm obscuring the IPs in the following excerpt from my xCAT 'hosts'
table, but the only real change in xCAT was that I defined a third ('C')
group of VMs that run on the HP cluster following the pattern I had done
for the previous 50+ VMs which were divided between the A and B groups:
"hpvmgroupA","|\D+(\d+).*$|X.Y.101.($1-0)|",,,"HP Compute nodes compute
interface",
"hpvmgroupB","|\D+(\d+).*$|X.Y.102.($1-0)|",,,"HP Compute nodes compute
interface",
"hpvmgroupC","|\D+(\d+).*$|X.Y.103.($1-0)|",,,"HP Compute nodes compute
interface",
Otherwise, if you do an lsdef and look at a node that was deployed
successfully which was part of hpvmgroupA and one that I am currently
trying to deploy in hpvmgroupC they are identical except for the details
that should differ like IP address and other things; and the osimage for
the Centos 6.5 image (which points to the *.tmpl kickstart files and the
otherpkgs *.tmpl files) is the same. So perhaps something else in my
environment changed?
My coworker and I have tried delving into the code and placing echoes as
debugging statements, and have looked at the -V verbose output of various
commands, but can't seem to find a meaningful error as to why it is not
fetching either the id_rsa or id_rsa.pub files. Perhaps this was taken
care of in a certain postscript that I have taken for granted before and
forgotten to run this time? I have no idea.
-Josh
On Tue, Mar 8, 2016 at 10:34 PM, Daniel Letai <[email protected]> wrote:
Can you confirm you are deploying stateful nodes and not stateless ?
On 03/09/2016 12:53 AM, Josh Nielsen wrote:
My coworker just pointed out that the /xcatpost/mypostscript on
the nodes that are deployed actually have this line:
ENABLESSHBETWEENNODES='NO'
export ENABLESSHBETWEENNODES
That's interesting, given that sshbetweennodes (without 'enable'
as the beginning of the parameter name?) was not defined at all in
the site table and the default is supposedly enabled(?). However,
I just set sshbetweennodes in site to
"sshbetweennodes","ALLGROUPS",, and am now redeploying to see if
it makes a difference.
The man page for site says:
sshbetweennodes: Comma separated list of groups to enable
passwordless root
ssh during install, or xdsh
-K. Default is ALLGROUPS.
Set to NOGROUPS,if you do not
wish to enabled any groups.
Service Nodes are not affected
by this attribute
they are always setup with
passwordless root access to
nodes and other SN.
-Josh
On Tue, Mar 8, 2016 at 4:26 PM, Josh Nielsen <
[email protected]> wrote:
Here is what I see in /var/log/messages when remoteshell is run:
Mar 7 14:28:41 xcat-serv1 node0087c xcat: remoteshell:
setup /etc/ssh/sshd_config and ssh_config
Mar 7 14:28:41 xcat-serv1 node0087c xcat: Install: setup
root .ssh
Mar 7 14:28:42 xcat-serv1 xcat-serv1 xCAT[16025]: xCAT:
Allowing getcredentials ssh_dsa_hostkey from node0087c
Mar 7 14:28:42 xcat-serv1 node0087c xCAT: remoteshell: getting
ssh_host_dsa_key
Mar 7 14:28:42 xcat-serv1 xcat-serv1 xCAT[16027]: xCAT:
Allowing getcredentials ssh_rsa_hostkey from node0087c
Mar 7 14:28:42 xcat-serv1 node0087c xCAT: ssh_rsa_hostkey
Mar 7 14:28:42 xcat-serv1 node0087c xCAT: start up sshd
I see new timestamps on authorized_keys and copy.sh when it is
run, so it is actually doing something.
Is there a substantial difference between remoteshell and
updatenode -k? Why does updatenode -k successfully copy the
id_rsa key to the node if I type in the password?
As for the hostname in /etc/sysconfig/network, actually if I run
just the kickstart and remove ifcfg-eth from the list of
postscripts (in the node definition's postscripts= field) to
execute automatically it ends up looking like this:
# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=3(NXDOMAIN)
But when I manually run updatenode node0087c -P ifcfg-eth it
works correctly. (Also it changes
the /etc/sysconfig/network-scripts/ifcfg-eth0 file's BOOTPROTO
parameter from dhcp to static and sets the IPADDR, as it
should).
# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=node0087c.morgan.haib.org
Only if I keep the ifcfg-eth postscript in the node definition
(listed in postscripts= if you 'lsdef') to be automatically
executed does that /etc/sysconfig/network file say 'localhost',
which seems to indicate an order of execution problem to me
(even though I made sure ifcfg-eth was listed last). And, to
answer your question, an nslookup node0087c before and after
both return the correct IP from either of the SNs' slave DNS
servers.
They are possibly two unrelated issues, but I'm close to just
upgrading xCAT and seeing if I have any better luck if I can't
figure out some obvious problem soon. For kicks I'll explicitly
set sshbetweennodes in the site table and rerun the remoteshell
postscript.
Regards,
Josh
On Tue, Mar 8, 2016 at 3:33 PM, Casandra H Qiu <
[email protected]> wrote:
mmm, I don't have system with xCAT 2.8.3. but I think
sshbetweenodes attribute is available for while. if it is not
defined in the site table, the default should be set up
passwordless between nodes.
"nslookup nodename" still works after you update the hostname,
right?
can u able to find any error message from logs? maybe in
the /var/log/message.
Thanks,
Casandra
...................................................................
Casandra Hong Qiu
Phone: (845) 433-9291, t/l 293-9291
Office: B/002, Floor 3, Z13
[email protected]
Inactive hide details for Josh Nielsen
---03/08/2016 03:58:33 PM---Thanks for the
response Casandra. I should firstly note
thatJosh Nielsen ---03/08/2016 03:58:33 PM---Thanks for the
response Casandra. I should firstly note that I have xCAT
2.8.3. I know I need to upg
From: Josh Nielsen <[email protected]>
To: xCAT Users Mailing list <[email protected]>
Date: 03/08/2016 03:58 PM
Subject: Re: [xcat-user] Updatenode -k won't create id_rsa key
without prompting for password
Thanks for the response Casandra. I should firstly note that I
have xCAT 2.8.3. I know I need to upgrade, but not only has
this worked in the past but I also successfully deployed 50+
nodes back in November with the exact same xCAT version I have
now and using the same osimage for Centos 6.5, same kickstart,
same defined postscripts, etc. So something else has changed,
perhaps in our environment?
That being said, I did not see sshbetweennodes specified at all
in the site table. The following are the only two references to
ssh in the table:
#tabdump site | grep -i ssh
"maxssh","8",,
"rsh","/usr/bin/ssh",,
Is 'sshbetweennodes' only a feature of versions newer than
2.8.X, or has it been around a while?
Lastly, you said that remoteshell copies over id_rsa.pub
(regardless - in either scenario), and I have seen that before
as well, but actually I am not seeing any id_rsa* keys (public
or private) copied to the node at all. And even an updatenode
-k is only producing the id_rsa (if I manually type the
password) but not the .pub, which is also odd. But
authorized_keys is populated with the rsa public key signature.
Something else must be going on.
P.S. The only other issue I'm still dealing with, which may
irrelevant for this issue, is a hostname problem to where if I
run the ifcfg-eth postscript it updates the hostname
in /etc/sysconfig/network from the correct node name to
"localhost". My forward and reverse lookup entries in DNS are
present, and the hostname is set correctly by the kickstart
before ifcfg-eth is run, and remains there if it is not run,
which I presume it gets from either the node definition in
dhcpd.leases which is created with 'makedhcp' and/or the DNS
entries for the host's IP. On the off chance that key copying
could be tied to name resolution inconsistencies I thought I
might mention that as well.
Thanks,
Josh
On Tue, Mar 8, 2016 at 1:20 PM, Casandra H Qiu <
[email protected]> wrote:
can u check the site table if sshbetweennodes is set up?
The default for sshbetweennodes is ALLGROUPS, and will
enable passwordless between nodes. this attribute will be
ignored if zone table is set up, so please check zone
table also.
if it enables, the remoteshell postscript will copy
id_rsa and id_rsa.pub over to compute node, otherwise, it
only copies id_rsa.pub.
from source code, updatenode -k is always required
password.
Thanks,
Casandra
...................................................................
Casandra Hong Qiu
Phone: (845) 433-9291, t/l 293-9291
Office: B/002, Floor 3, Z13
[email protected]
Inactive hide details for Josh Nielsen
---03/08/2016 12:51:58 PM---Yes, I just
verified. It is present, but that alone is
not sJosh Nielsen ---03/08/2016 12:51:58 PM---Yes, I just
verified. It is present, but that alone is not sufficient
for that node to be able to SS
From: Josh Nielsen <[email protected]>
To: xCAT Users Mailing list <
[email protected]>
Date: 03/08/2016 12:51 PM
Subject: Re: [xcat-user] Updatenode -k won't create
id_rsa key without prompting for password
Yes, I just verified. It is present, but that alone is
not sufficient for that node to be able to SSH to other
nodes itself. It allows other nodes which have the
correct private key to SSH to it, but not the other way
around.
For example, on one compute node I'm having trouble
with /root/.ssh has these three files:
-rw-------. 1 root root 408 Mar 7 14:28 authorized_keys
-rw-------. 1 root root 411 Mar 7 14:28 copy.sh
-rw------- 1 root root 402 Mar 3 16:20 known_hosts
And authorized_keys has the correct ssh-rsa public key
entry, but I cannot go from this node to any other node
in my cluster via passwordless ssh. But as soon as I run
updatenode -k, and type in the password that it prompts
for to complete the command, the id_rsa key is added as
the fourth file to the /root/.ssh directory, and then
after that I can ssh to other nodes from it without
supplying a password. That is the issue.
In the past simply running the remoteshell postscript (or
so I assumed) was sufficient for adding the id_rsa file,
and it was all automated from a fresh deploy by
specifying remoteshell as one of the default postscripts
to run. But now it doesn't look like remoteshell is
placing the id_rsa file on the node (unless some other
script or command is responsible for that), but
remoteshell looks like it creates everything else
in /root/.ssh/ (and /etc/ssh/).
Is remoteshell the correct postscript for that, or was
the id_rsa key most likely being pushed to the nodes some
other way (like by some code that called updatenode -k
upon initial deployment)? Either way, all I can say for
sure is that id_rsa used to appear in /root/.ssh on the
compute node automatically and now it does not.
Regards,
Josh
On Tue, Mar 8, 2016 at 4:19 AM, Xiao Peng Wang <
[email protected]> wrote:
To enable the login without password, the rsa
public key should be copied
to /root/.ssh/authorized_keys in the compute
node. Could you check whether the key has
been added in to
/root/.ssh/authorized_keys?
Thanks
Best Regards
----------------------------------------------------------------------
Wang Xiaopeng (王晓朋)
IBM China System Technology Laboratory
Tel: 86-10-82453455
Email: [email protected]
Address: 28,ZhongGuanCun Software Park,No.8
Dong Bei Wang West Road, Haidian District
Beijing P.R.China 100193
----- Original message -----
From: Josh Nielsen <[email protected]>
To: xCAT Users Mailing list <
[email protected]>
Cc:
Subject: Re: [xcat-user] Updatenode -k won't
create id_rsa key without prompting for
password
Date: Tue, Mar 8, 2016 5:26 AM
Also if remoteshell is invoked directly as a
postscript ('updatenode node0086c -V -P
remoteshell') it produces the same result,
but does not prompt for a password (like
invoking xdsh -K directly doesn't), and
copies everything over except id_rsa. So
actually the prompting for a password is
specific to updatenode -k, not xdsh -K or the
remoteshell postscript (which run that). So
I'm not sure if that is relevant to the
underlying problem or not, but if I do
invoke updatenode -k and supply it the
password it copies the id_rsa to the node.
On Mon, Mar 7, 2016 at 2:12 PM, Josh Nielsen
<[email protected]> wrote:
Hello,
When we freshly deploy a node from the
kickstart and run our postscripts we noticed
that for some reason the /root/.ssh/id_rsa
file which allows passwordless login from
that node to other nodes is missing, though
this was not the case just a few months
ago. When I try to generate the key manually
it prompts for a password, after which it
will copy/create that file successfully (see
below), but there are a few odd things
connected to this.
The error is:
updatenode node0087c -k
Enter the password for the userid: root on
the node where the ssh keys will be updated:
The first oddity is that even after supplying
the password once for a particular node it
will prompt for the password every time if I
run it again, as well as the related problem
that this never used to happen before and the
key used to be created without issue or
prompting for a password. The 'passwd' xCAT
table has the password for root (if that is
where it looks for this command).
Secondly I have done several manual debugging
steps (and poking around source code to see
what is happening) and I have run the actual
xdsh command that is called, shown from the
-V verbose output (which it prints two of,
the first apparently to prep the SNs and run
the 'remoteshell' postscript on them, and the
second to actually do the same to the node
specified).
xdsh sn1,sn2 --nodestatus -s -v
-e /install/postscripts/xcatdsklspost 5 -m
[MN_IP] 'remoteshell,servicenode'
--tftp /tftpboot --installdir /install
--nfsv4 no -c -V
xdsh node0086c --nodestatus -s -v
-e /install/postscripts/xcatdsklspost 5 -m
[SN1_IP] 'remoteshell' --tftp /tftpboot
--installdir /install --nfsv4 no -c -V
This did not reveal anything useful, except
that when invoked directly like this no
password is prompted for and it runs, but
still leaves out the id_rsa file. I followed
also the suggestion by Wang Xaiopeng in this
thread (http://tinyurl.com/jz2jzmb) to test
the getcredentials call with:
1. Enable mini server
/xcatpost/allowcred.awk &
2.Try to get rsa hostkey
USEOPENSSLFORXCAT=yes
XCATSERVER=<MNIP>:3001
/xcatpost/getcredentials.awk
ssh_rsa_hostkey
This returned ssh_rsa_hostkey sucessfully.
When remoteshell is run (whether with
updatenode -k or xdsh -K) it actually does
copy over the key files into /etc/ssh/ and it
copies known_hosts, copy.sh, and
authorized_keys into /root/.ssh on the
compute node but omits id_rsa. What could be
going wrong here?
Regards,
Josh Nielsen
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications
with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications
with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
