On Sun, Nov 9, 2008 at 7:54 PM, Jeff Victor <[EMAIL PROTECTED]> wrote:
> It has become clear that there is a need to monitor resource
> consumption of workloads in zones, and an easy method to compare
> consumption to resource controls. In order to understand how a
> software tool could fulfill this need, I created an OpenSolaris
> project and a prototype to get started. If this sounds interesting,
> you can find the project and Perl script at:
> http://opensolaris.org/os/project/zonestat/ .
> If you have any comments, or suggestions for improvement, please let
> me know on this e-mail list or via private e-mail.

I've had such needs for a while and have developed some tools to help
my organization with that.  Unfortunately, I'm not able to share that
code.  I am able to share suggestions...

I am in a habit of:

#! /usr/bin/perl -w

use strict;

That catches a lot of mistakes that may be masked by:

close STDERR;

which I never do. :)

Please do not use /etc/release as a test of kernel functionality.
Those that patch to an equivalent level as the update release have a
similar level of functionality.  A better mechanism would be to check
for specific kernel patches.  See
for kernel patch ID's through Update 6.  The blog entry was posted
just before U5 shipped, so the U6 info should be checked for accuracy.

# Get amount and cap of memory locked by processes in each zone.
# "kstat -p caps:*:lockedmem_zone_*" conveniently summarizes all zones for us.
open(KSTAT, "/usr/bin/kstat -p caps:*:lockedmem_zone_* |");
while (<KSTAT>) {

You could just use Sun::Solaris::Kstat rather than forking another perl script.

My feeling on capped memory is that if it becomes an issue and capped
swap is not really close to capped memory, the over-consumptive zone
has too high of a chance of causing horrible I/O problems for all
zones.  That is, the cap is likely to do more harm than good.  This
may change if swap can go onto solid state disk.  I only mention this,
because I don't see a purpose in capping RSS, rather I cap swap.
FWIW, I tend to use the term "reserved memory" instead of "swap"
because that is less confusing to most people.

For CPU related stats, take a look at a discussion I started a while back:


One project I would like to kick off sometime is doing per user, per
project, and per zone microstate accounting.  Presumably this data
would be available through kstat.  The tricky part here is to not
introduce a big load on the system in the process of doing this.  The
above discussion and/or others in a similar vein have led me to think
that collecting stats as processes exit and periodically through a
kernel thread would be the way to go.  This approach won't be accurate
to subsecond intervals, but generally speaking you don't need better
data than per minute.  Such a thread should have no more impact on the
system than a single user running prstat or top with a similar
interval.  Further, it would be good data for prstat (e.g. -a) to use.

A follow-on to that would be to have a way to track usage of kernel
taskq work.  As more in-kernel functionality comes into existence, it
becomes harder to see where the utilization is.  For example, a kstat
that counted the relative amount of time in zfs crypto versus zfs
gzip9 operations would be helpful to the support person that is trying
to answer the call "why does vmstat say my system is pegged while
prstat shows no processes consuming CPU?"  Sure, dtrace can get that
information - but it is hard for the typical person to write and
pretty expensive to run as regular monitoring.  But, now I'm a bit off

The use of mdb rules out use by most users.  I dislike tools that make
users (e.g. application owners) ask me for root access.  I haven't
looked closely to see which, if any, of the other commands also
require some elevated privileges.

Most (all?) other commands have the full path set but mdb doesn't.
Perhaps $ENV{PATH} = ... would be a good thing to add.

I didn't have a chance to check logic closely or run it on a test
system.  I'll offer more feedback if needed when I get a chance to
test it.  It is a great start and I can't wait to see it progress.

Mike Gerdts
zones-discuss mailing list

Reply via email to