On Wed, 13 Jul 2011 at 12:18 -0000, Denis Demidov wrote:
> Here (https://github.com/ddemidov/liqueur) is an attempt to make
> small and simple python script for powering nodes of gridengine
> cluster up or down depending on queue state. The script is inspired
> by SPIRIT (http://www.ciul.ul.pt/~ATP/SPIRIT). The main difference
> is that queue status is parsed from xml output of qhost and qstat.
> Also, nodes are shut down not immediately, but after some predefined
> interval.
Your code looks much cleaner than the Spirit code (C isn't really a
good language for scripting).
Look at the "Green Computing" thread from April 29, 2011. There are
some good comments in the follow ups.
I'm working on some similar functionality and expect to make it
available once it is closer to complete. I don't expect it to be
fully general, there are all sort of small cases to worry about.
I'm using IPMI for system shutdown and start up (I may switch to ssh
for power down due to some issues I'm currently seeing).
I'm using a cluster queue of "green" nodes doing power on/off to keep
a certain range of these nodes idle. This doesn't work if there is a
pending MPI job which needs more nodes than are currently powered on.
I may need to look at the pending job queue in the future, at the
moment large MPI isn't a big issue on this cluster.
Keeping a small number of idle nodes available speeds up ramp up
process when jobs are submitted. Our nodes take several minutes to
boot up.
I also go through a two phase shutdown process where I first disable
the node to ensure SGE doesn't try to start a job just as I start a
shutdown. I also don't enable the node until it comes back up after
power on. I'll look at skipping this step once things are working
usefully.
I expect getting the hysteresis right to be a tuning exercise between:
- responsiveness when jobs are queued
- node start up time requirements
- limiting hard power cycles on nodes
- energy savings
My processing of the XML output is much more complicated than yours.
I probably don't know the best way to process XML in perl.
Stuart
--
I've never been lost; I was once bewildered for three days, but never lost!
-- Daniel Boone
_______________________________________________
users mailing list
[email protected]
https://gridengine.org/mailman/listinfo/users
