Just as a follow up to this thread. I spoke with someone from IBM and they tell me that 2 cores of 4 hardware threads each are hidden from the kernel (how do they do that?) and used for the custom HS4 cards we have installed on the IO nodes, which explains why I see only 60 instead of 68 threads. the 2 bgvrnic tasks I see spinning at 100% run on threads 58/59 and service the connection from ION to CN.
It looks as though everything is reporting as expected - as long as I compile hwloc on the ION itself, it seems to be correct. Thanks and sorry for any misunderstanding JB > -----Original Message----- > From: hwloc-users [mailto:hwloc-users-boun...@open-mpi.org] On Behalf > Of Chris Samuel > Sent: 26 March 2014 13:42 > To: Hardware locality user list > Subject: Re: [hwloc-users] BGQ question. > > On Wed, 26 Mar 2014 11:56:08 AM Biddiscombe, John A. wrote: > > > I can’t test this as the system is down for maintenance, but if memory > > serves me correctly, the GCC compiled lstopo also showed 60 cores > > instead of 64/68. > > It can only report what the kernel reports and it appears your kernel is not > reporting the same number of cores on an IO node as ours. > > It would be interesting to compare kernel version and boot command line. > > Ours are: > > -bash-4.1# uname -a > Linux r00-id-j01.pcf.vlsci.unimelb.edu.au 2.6.32- > 279.14.1.bgq.el6_V1R2M0_36.ppc64 #1 SMP Tue Jun 11 15:50:53 CDT 2013 > ppc64 ppc64 ppc64 GNU/Linux > > > -bash-4.1# cat /proc/cmdline > root=/dev/ram0 rdinit=/init raid=noautodetect loglevel=5 > > > This is the end of our /proc/cpuinfo showing 68 hardware threads > (17 cores exposed). > > -bash-4.1# tail -n 9 /proc/cpuinfo > > processor : 67 > cpu : A2 (Blue Gene/Q) > clock : 1600.000000MHz > revision : 2.0 (pvr 0049 0200) > > timebase : 1600000000 > platform : Blue Gene/Q > model : ibm,bluegeneq > > > > I am not certain if this gcc was in any was ‘special’ for bgq. > > There is a GCC cross compiler, but it's not the /usr/bin/gcc one. > > cheers! > Chris > -- > Christopher Samuel Senior Systems Administrator > VLSCI - Victorian Life Sciences Computation Initiative > Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545 > http://www.vlsci.org.au/ http://twitter.com/vlsci > > _______________________________________________ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
