On Wed, 6 Feb 2013, Orlando Richards wrote:
...
I've had a go at digging through the code, but couldn't really make head
nor tail of it - no doubt in large part due to my not being much of a
coder :( Any pointers to get me bootstrapped would be most welcome.
General comments about the source...
Don't be intimidated. It's a large code base, but spend a little time and
it'll start to make sense. Pick a little bit of it to focus on initially.
Gridengine's source code is layered. The source distribution has a few
HTML files describing them (some of which still need updating from the 6.0
days...). Functions near the very top and very bottom of the stack are
relatively well commented, but the rest can be a little hit and miss.
Ignoring most of the layers, you've essentially got:
At the bottom you've got the wonderful CULL layer: it's very solid and
provides gridengine with safe complicated data structures. I'd like to pat
the person who wrote it on the back, although I admit I've yet to get my
head round the advanced search functionality. State data for jobs and so
on tend to use it. Use of it can be identified by the data types or
functions prefixed with "l".
While I'm on data structures, there are also "dstrings" - which provide
safe string handling.
In the middle you've got the GDI, which is the set of libraries used by
the different components to communicate with each other over the network.
At the top you've got the qmaster, execd, etc., which can be thought of as
loosely coupled applications that all use the same underlying
libraries/layers to coordinate.
I've spent most of my time in the execd, which is pretty easy but messy [a
very large number of special cases - not totally unexpected with the
number of platforms supported over the years, but ripe for some
refactoring]. I've had a brief play in the qmaster and my first impression
is that it's more consistent and "solid" than the execd, but more
complicated.
General tips for debugging gridengine...
1) Play with the loglevel setting in "qconf -sconf" and read the messages
files.
2) Figure out how to stick gridengine into debug mode.
https://blogs.oracle.com/templedf/entry/using_debugging_output
Essentially something like:
* Setup sge environment (SGE_ROOT, SGE_QMASTER_PORT, etc.)
* Execute: . $SGE_ROOT/util/dl.sh
* Execute: dl 1
* Execute: $SGE_ROOT/bin/lx-amd64/sge_execd
The program will not daemonise and will print lots of interesting stuff.
Different 'dl' values will give you different output. I generally find
that anything greater than 1 is "too much".
This technique will work for pretty much any gridengine component. Even
qsub.
3) Run gridengine under gdb.
I don't know if you've had much experience with gdb but, once you've got
the hang of it, it's very useful in figuring out what some code generally
does without actually understanding the details. Once you've followed your
nose to something that doesn't look right, you can then spend time
figuring things out.
I think some of the gridengine forks try to provide builds with enough
debugging information for this to work, but I tend to build my own
gridengine so that I can easily recompile after editing the source with
potential fixes.
Make sure you build with the "-no-opt" and "-debug" flags to aimk
(disables optimisation and enables debugging symbols) and keep the source
tree kicking around for gdb to read. I run our production gridengine with
those flags and haven't noticed any serious performance problems.
Once you have gridengine running under gdb and playing with breakpoints
and the rest, you can easily examine interesting data structures with
commands like "p lWriteList(ptr)", "p lWriteElem(ptr)" and
"p sge_dstring_get_string(ptr)" (where ptr is a lList*, lListElem* or
dstring*, respectively).
...
At the moment, I'm trying to get a reproducible test case together to
allow for useful debugging - basic tests (sleep 60s) don't show an
obvious triggering of the issue, so I'm moving onto more complicated
tasks. Certainly, the issue does seem to create orders-of-magnitude
differences in reported usage. Current offenders include BLAST jobs (run
by our Biology users) - which are fairly memory heavy.
...
Being able to reproduce the problem will obviously make things far, far
easier! If you cannot, you're probably reduced to littering the relevant
qmaster code with INFO(())/WARNING(())/ERROR(()) statements (and checking
that loglevel in "qconf -sconf" is set to the appropriate value) and
seeing what appears in the messages files in production.
If you're lucky, the problem might be evident in the usage information
being sent from the execd to the qmaster. Running the execd in debug mode
with "dl 1" will reveal what CPU/MEM/IO values the qmaster is being given
to be used in the accounting file and the share tree.
If you're unlucky, the problem is in how the qmaster aggregates, records
and decays the share tree values over time.
If you're really unlucky, the problem might only occur if the various
gridengine components are under severe stress.
I find that having a non-production installation of gridengine kicking
around, perhaps in virtual machines, is very handy :)
Hope this helps...
Mark
--
-----------------------------------------------------------------
Mark Dixon Email : [email protected]
HPC/Grid Systems Support Tel (int): 35429
Information Systems Services Tel (ext): +44(0)113 343 5429
University of Leeds, LS2 9JT, UK
-----------------------------------------------------------------
_______________________________________________
users mailing list
[email protected]
https://gridengine.org/mailman/listinfo/users
