Lissa/Jarrod,
As I suspected, there was a corruption of some type in the xCAT SQLite
files. I made a backup of /etc/xcat before I upgraded from 2.8.3 to 2.8.4. So,
I stopped xCATd, deleted /etc/xcat, restored it from backup created at 2.8.3
and then started xCATd back up. When I did, xCATd re-ran the schema updates and
now everything is working correctly. Not sure if this is something in the RPM
post section that messed up, or if the RPM ran the Schema updates without
stopping xcatd, or what.
I have copies of my 2.8.3 database and 2.8.4 (post RPM that was broken) and
2.8.4 (working), if either of you are interested in debugging the problems I
experienced, I can bundle them up and upload them to you in some secure fashion.
Just one more reason to make sure you have a complete and correct backup
before performing any updates :).
However, that doesn't fix the problem of nodediscoverls reporting newly
discovered nodes via SNMP as "undef" instead of "switch". It's still doing
that, doesn't seem to have an impact on anything, but I am just letting you
know in case it might impact some other piece of xCAT.
Thanks,
-Brad Viviano
===================================================
Brad Viviano
High Performance Computing & Scientific Visualization
Lockheed Martin, Supporting the EPA
Research Triangle Park, NC
919-541-2696
HSCSS Task Order Lead - Ravi Nair
919-541-5467 - nair.r...@epa.gov
High Performance Computing Subtask Lead - Durward Jones
919-541-5043 - jones.durw...@epa.gov
Environmental Modeling and Visualization Lead - Heidi Paulsen
919-541-1834 - paulsen.he...@epa.gov
________________________________
From: Lissa Valletta <lis...@us.ibm.com>
Sent: Monday, July 28, 2014 12:18 PM
To: xCAT Users Mailing list
Subject: Re: [xcat-user] Upgraded to xCAT 2.8.4, nodeset not working as
expected.
All you have to do to restore your database is run restorexCATdb -p <path to
you backup directory>. It will replace all the tables. You should never
remove any of the files manually and you do not have to stop xcatdb. But
since it looks like you already have removed files, do the restore in bypass
mode.
service xcatd stop. ps -ef | grep xcatd and make sure all processes are gone.
export XCATBYPASS=y
restorexCATdb -p <path to you backup>
unset XCATBYPASS
service xcatd start
Lissa K. Valletta
8-3/B10
Poughkeepsie, NY 12601
(tie 293) 433-3102
[Inactive hide details for "Viviano, Brad" ---07/28/2014 11:45:28 AM---Jarrod,
%bphash isn't coming into xnba.pm::setstate a]"Viviano, Brad" ---07/28/2014
11:45:28 AM---Jarrod, %bphash isn't coming into xnba.pm::setstate as
expected. Data::Dumper shows:
From: "Viviano, Brad" <viviano.b...@epa.gov>
To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
Date: 07/28/2014 11:45 AM
Subject: Re: [xcat-user] Upgraded to xCAT 2.8.4, nodeset not working as
expected.
________________________________
Jarrod,
%bphash isn't coming into xnba.pm::setstate as expected. Data::Dumper shows:
BPHash: $VAR1 = 'c4u34';
$VAR2 = [
undef
];
Compared to my working 2.8.4 cluster:
BPHash: $VAR1 = 'c0u13';
$VAR2 = [
{
'kcmdline' =>
'imgurl=http://172.20.0.1:80//install/netboot/rhel6_c/x86_64/compute/rootimg.gz
XCAT=!myipfn!:3001 NODE=c0u13 FC=0 console=tty0 console=ttyS0,115200',
'kernel' => 'xcat/osimage/rhel6_c-x86_64-netboot-compute/kernel',
'initrd' =>
'xcat/osimage/rhel6_c-x86_64-netboot-compute/initrd-stateless.gz'
}
];
So, it feels like I have a corruption in the database, maybe a hash or key is
screwed up causing the hang and problems. I have a backup of the database files
from dumpxCATdb. What's the best way to clear out and restore the database
files from the backup.
I tried to stop xcatd, remove all the /etc/xcat/*.sqlite files and restart
xcatd, but it complains. What is the approved procedure to restore sqlite files
from backup?
-Brad
===================================================
Brad Viviano
High Performance Computing & Scientific Visualization
Lockheed Martin, Supporting the EPA
Research Triangle Park, NC
919-541-2696
HSCSS Task Order Lead - Ravi Nair
919-541-5467 - nair.r...@epa.gov
High Performance Computing Subtask Lead - Durward Jones
919-541-5043 - jones.durw...@epa.gov
Environmental Modeling and Visualization Lead - Heidi Paulsen
919-541-1834 - paulsen.he...@epa.gov
________________________________
From: Viviano, Brad <viviano.b...@epa.gov>
Sent: Monday, July 28, 2014 10:50 AM
To: xCAT Users Mailing list
Subject: Re: [xcat-user] Upgraded to xCAT 2.8.4, nodeset not working as
expected.
Just what I would expect:
# date ; /usr/bin/time -p nodeset c4u34 osimage=rhel6_c-x86_64-netboot-compute
Mon Jul 28 10:44:49 EDT 2014
c4u34: netboot rhel6_c-x86_64-compute
real 31.77
user 0.11
sys 0.03
>From /var/log/messages:
Jul 28 10:44:50 sol-mgmt xCAT[16624]: xCAT: Allowing nodeset to c4u34
osimage=rhel6_c-x86_64-netboot-compute for root from localhost
The times are right on. I am looking through the xnba.pm module now. The value
that is getting set:
# cat /tftpboot/xcat/xnba/nodes/c4u34
#!gpxe
#netboot rhel6_c-x86_64-compute
LOCALBOOT 0
is the DEFAULT case in the IF statement. I am going to start putting in some
comment lines to see where it's hanging in the code. Maybe there is something
wrong with xcatd and the sqlite database files that's causing things to hang or
return incorrect data.
Any suggestions on what I should be looking for would be helpful.
-Brad
===================================================
Brad Viviano
High Performance Computing & Scientific Visualization
Lockheed Martin, Supporting the EPA
Research Triangle Park, NC
919-541-2696
HSCSS Task Order Lead - Ravi Nair
919-541-5467 - nair.r...@epa.gov
High Performance Computing Subtask Lead - Durward Jones
919-541-5043 - jones.durw...@epa.gov
Environmental Modeling and Visualization Lead - Heidi Paulsen
919-541-1834 - paulsen.he...@epa.gov
________________________________
From: Jarrod B Johnson <jbjoh...@us.ibm.com>
Sent: Monday, July 28, 2014 10:37 AM
To: xCAT Users Mailing list
Subject: Re: [xcat-user] Upgraded to xCAT 2.8.4, nodeset not working as
expected.
Does anything conspicous appear in /var/log/messages?
[Inactive hide details for "Viviano, Brad" ---07/28/2014 10:22:58 AM---Jarrod,
Sorry. The rsetboot isn't taking a long time.]"Viviano, Brad" ---07/28/2014
10:22:58 AM---Jarrod, Sorry. The rsetboot isn't taking a long time. It's
just nodeset that is. I cut/pasted a
From: "Viviano, Brad" <viviano.b...@epa.gov>
To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
Date: 07/28/2014 10:22 AM
Subject: Re: [xcat-user] Upgraded to xCAT 2.8.4, nodeset not working as
expected.
________________________________
Jarrod,
Sorry. The rsetboot isn't taking a long time. It's just nodeset that is. I
cut/pasted a group of commands into the terminal for the system I was working
on (nodeset, rsetboot, rpower) and it hung on nodeset. So, there is something
going wrong with nodeset. I have another xCAT iDataplex cluster (my testing
cluster) which I upgraded to 2.8.4 and it isn't having a problem:
# cat /tftpboot/xcat/xnba/nodes/c0u13
#!gpxe
#netboot rhel6_c-x86_64-compute
imgfetch -n kernel
http://${next-server}/tftpboot/xcat/osimage/rhel6_c-x86_64-netboot-compute/kernel
imgload kernel
imgargs kernel
imgurl=http://172.20.0.1:80//install/netboot/rhel6_c/x86_64/compute/rootimg.gz
XCAT=${next-server}:3001 NODE=c0u13 FC=0 console=tty0 console=ttyS0,115200
BOOTIF=01-${netX/machyp}
imgfetch
http://${next-server}/tftpboot/xcat/osimage/rhel6_c-x86_64-netboot-compute/initrd-stateless.gz
imgexec kernel
# /usr/bin/time -p nodeset c0u13 boot
c0u13: boot
real 0.47
user 0.07
sys 0.01
# cat /tftpboot/xcat/xnba/nodes/c0u13
#!gpxe
#boot
exit
# /usr/bin/time -p nodeset c0u13 osimage=rhel6_c-x86_64-netboot-compute
c0u13: netboot rhel6_c-x86_64-compute
real 1.01
user 0.07
sys 0.03
# cat /tftpboot/xcat/xnba/nodes/c0u13
#!gpxe
#netboot rhel6_c-x86_64-compute
imgfetch -n kernel
http://${next-server}/tftpboot/xcat/osimage/rhel6_c-x86_64-netboot-compute/kernel
imgload kernel
imgargs kernel
imgurl=http://172.20.0.1:80//install/netboot/rhel6_c/x86_64/compute/rootimg.gz
XCAT=${next-server}:3001 NODE=c0u13 FC=0 console=tty0 console=ttyS0,115200
BOOTIF=01-${netX/machyp}
imgfetch
http://${next-server}/tftpboot/xcat/osimage/rhel6_c-x86_64-netboot-compute/initrd-stateless.gz
imgexec kernel
Granted my test cluster is only a few nodes. But still, it's updating
everything correctly. I also noticed that the node I put through discovery on
2.8.4 showed discovery method as "UNDEF" according to nodediscoverls:
# nodediscoverls | grep c4u34
18F64EE1-3CC1-DF11-B526-E41F134D13C2 c4u34 undef
7323AC1 23DF198
Where are my other nodes show it as "switch":
# nodediscoverls | grep c4u33
0ED7839B-3FC1-DF11-B752-E41F134D42D8 c4u33 switch
7323AC1 23DF167
-Brad
===================================================
Brad Viviano
High Performance Computing & Scientific Visualization
Lockheed Martin, Supporting the EPA
Research Triangle Park, NC
919-541-2696
HSCSS Task Order Lead - Ravi Nair
919-541-5467 - nair.r...@epa.gov
High Performance Computing Subtask Lead - Durward Jones
919-541-5043 - jones.durw...@epa.gov
Environmental Modeling and Visualization Lead - Heidi Paulsen
919-541-1834 - paulsen.he...@epa.gov
________________________________
From: Jarrod B Johnson <jbjoh...@us.ibm.com>
Sent: Monday, July 28, 2014 9:53 AM
To: xCAT Users Mailing list
Subject: Re: [xcat-user] Upgraded to xCAT 2.8.4, nodeset not working as
expected.
Odd that rsetboot should take a long time. I assume something like rpower stat
also takes a long time?
[Inactive hide details for "Viviano, Brad" ---07/28/2014 09:49:38 AM---Good
morning, I upgraded xCAT via yum last week from]"Viviano, Brad"
---07/28/2014 09:49:38 AM---Good morning, I upgraded xCAT via yum last week
from 2.8.3 to to 2.8.4. Since then, if I put a n
From: "Viviano, Brad" <viviano.b...@epa.gov>
To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
Date: 07/28/2014 09:49 AM
Subject: [xcat-user] Upgraded to xCAT 2.8.4, nodeset not working as expected.
________________________________
Good morning,
I upgraded xCAT via yum last week from 2.8.3 to to 2.8.4. Since then, if I
put a node (in this case a dx360M3) through discovery, discovery completes, but
if I then try and use nodeset to use toolscenter or set this diskless image on
the node (nodeset XXX osimage=), it isn't working.
I'm noticing that the nodeset and rsetboot commands are taking > 30 seconds
to complete and when I reboot the node, it just tries to boot from disk. A
little digging and it appears that nodeset isn't updating the underlying files
on /tftpboot/xcat/xnba/nodes/* like expected:
# /usr/bin/time -p nodeset c4u34 netboot=toolscenter-x86_64-bomc
c4u34: netboot toolscenter-x86_64-bomc
real 31.20
user 0.11
sys 0.03
# cat /tftpboot/xcat/xnba/nodes/c4u34
#!gpxe
#netboot toolscenter-x86_64-bomc
LOCALBOOT 0
# /usr/bin/time -p nodeset c4u34 osimage=rhel6_c-x86_64-netboot-compute
c4u34: netboot rhel6_c-x86_64-compute
real 49.85
user 0.12
sys 0.01
# cat /tftpboot/xcat/xnba/nodes/c4u34
#!gpxe
#netboot rhel6_c-x86_64-compute
LOCALBOOT 0
The long time makes me think it's hitting a timeout value and aborting. Did I
miss something with the upgrade to 2.8.4? I have no problems with 2.8.3 and
discovery is working on 2.8.4.
Thanks,
-Brad Viviano
===================================================
Brad Viviano
High Performance Computing & Scientific Visualization
Lockheed Martin, Supporting the EPA
Research Triangle Park, NC
919-541-2696
HSCSS Task Order Lead - Ravi Nair
919-541-5467 - nair.r...@epa.gov
High Performance Computing Subtask Lead - Durward Jones
919-541-5043 - jones.durw...@epa.gov
Environmental Modeling and Visualization Lead - Heidi Paulsen
919-541-1834 -
paulsen.he...@epa.gov------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
