Ralph -- let's chat about this in Chicago next Friday. I'll add it to the
agenda on the wiki. I assume this would not be difficult stuff; we don't
really need to do anything fancy at all. I think we just want to sketch out
what exactly we want to do, and it could probably be done in a day or three.
(Thanks for the idea, Samuel!)
On Dec 4, 2013, at 7:49 AM, Ralph Castain <r...@open-mpi.org> wrote:
> FWIW: ORTE already has a sensor framework in it that reads some of these
> things, so adding the coretemp etc is pretty trivial. These readings can be
> taken in the ORTE event thread on daemons, but we could allow procs to do so
> as well (if the app requests it), or can make it driven via the MPI_T
> function. If we use the event engine, we could have ORTE push those values
> into the internal database, and then provide an MPI_T access to retrieve them.
>
> I'm working on the monitoring section over the next few weeks and can add the
> data collection part. If Jeff or someone can point me to the required MPI_T
> "glue", I can add that too.
>
>
> On Wed, Dec 4, 2013 at 4:23 AM, Chris Samuel <sam...@unimelb.edu.au> wrote:
> On Wed, 4 Dec 2013 11:39:29 AM Jeff Squyres wrote:
>
> > On Dec 3, 2013, at 7:54 PM,
> > Christopher Samuel <sam...@unimelb.edu.au> wrote:
> >
> > > Would it make any sense to expose system/environmental/thermal
> > > information to the application via MPI_T ?
> >
> > Hmm. Interesting idea.
>
> Phew. :-)
>
> > Is the best way to grab such stuff via IPMI?
>
> I don't think so, that means either having the process have permissions to
> access /dev/ipmi* or needing to talk over the network to the adapter, neither
> of which are likely to be desirable (or even possible, our iDataplex IMMs are
> not accessible from the compute nodes).
>
> However, using the coretemp kernel module means you get access to at least
> information about CPU temperatures on Intel systems:
>
> /sys/bus/platform/devices/coretemp.${A}/temp${B}_input
>
> which contains the core temperature in 100ths of a degree Celsius and are
> world readable. You also get access to the various thermal trip points and
> alarms.
>
> The ${B} value is 1 for the CPU package (SandyBridge or later only), then
> sequentially for the physical cores. ${A} is 0 for the first socket, then
> max($B of $A)+1 for the next socket, etc..
>
> So on the test login node of our 2010 era Nehalem iDataplex you get a file per
> CPU core but nothing for the socket, viz:
>
> [root@merri-test ~]# ls /sys/bus/platform/devices/coretemp.*/*input*
> /sys/bus/platform/devices/coretemp.0/temp2_input
> /sys/bus/platform/devices/coretemp.0/temp3_input
> /sys/bus/platform/devices/coretemp.0/temp4_input
> /sys/bus/platform/devices/coretemp.0/temp5_input
> /sys/bus/platform/devices/coretemp.4/temp2_input
> /sys/bus/platform/devices/coretemp.4/temp3_input
> /sys/bus/platform/devices/coretemp.4/temp4_input
> /sys/bus/platform/devices/coretemp.4/temp5_input
>
> [root@merri-test ~]# cat /sys/bus/platform/devices/coretemp.*/*input*
> 52000
> 52000
> 52000
> 53000
> 59000
> 55000
> 58000
> 56000
>
> On the test login node of our SandyBridge iDataplex delivered mid year we get
> the package as well:
>
> [root@barcoo-test ~]# ls /sys/bus/platform/devices/coretemp.*/*input*
> /sys/bus/platform/devices/coretemp.0/temp1_input
> /sys/bus/platform/devices/coretemp.0/temp2_input
> /sys/bus/platform/devices/coretemp.0/temp3_input
> /sys/bus/platform/devices/coretemp.0/temp4_input
> /sys/bus/platform/devices/coretemp.0/temp5_input
> /sys/bus/platform/devices/coretemp.0/temp6_input
> /sys/bus/platform/devices/coretemp.0/temp7_input
> /sys/bus/platform/devices/coretemp.6/temp1_input
> /sys/bus/platform/devices/coretemp.6/temp2_input
> /sys/bus/platform/devices/coretemp.6/temp3_input
> /sys/bus/platform/devices/coretemp.6/temp4_input
> /sys/bus/platform/devices/coretemp.6/temp5_input
> /sys/bus/platform/devices/coretemp.6/temp6_input
> /sys/bus/platform/devices/coretemp.6/temp7_input
>
> [root@barcoo-test ~]# cat /sys/bus/platform/devices/coretemp.*/*input*
> 44000
> 43000
> 44000
> 42000
> 43000
> 38000
> 44000
> 37000
> 33000
> 37000
> 32000
> 34000
> 36000
> 33000
>
> There's more information in $KERNEL_SOURCE/Documentation/hwmon/coretemp.
>
> Both those systems are running RHEL6, so it should be fairly well supported
> *if* the sysadmin has loaded the modules.
>
> > That might well be do-able, since there's no performance penalty for reading
> > such values until you actually read the values (i.e., we don't actively
> > monitor these values in OMPI's overall progression engine; they're only
> > read when the application invokes an MPI_T read function).
>
> Indeed, these *shouldn't* hang trying to read them. ;-)
>
> cheers,
> Chris
> --
> Christopher Samuel Senior Systems Administrator
> VLSCI - Victorian Life Sciences Computation Initiative
> Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
> http://www.vlsci.org.au/ http://twitter.com/vlsci
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
