Dan Rich wrote:
[EMAIL PROTECTED] wrote:I'm not sure I agree with your conclusion. Why couldn't the nagios plugin connect to a gmond on one of the nodes in each cluster and parse the XML. It should also have a timeout to go to another node, if the "primary" happens to be down. I see ganglia's purpose as to collect performance information, not monitor services. I think it makes more sense to keep these functions separate. However, I think writing nagios plugins to utilize ganglia information makes sense. --Shane [EMAIL PROTECTED] said:Sadly, Nagios doesn't play well with Ganglia. See my (ancient) posting to the Beowulf list:The idea above is pretty-much what I was considering. I already have my nagios server receiving multicast from my various clusters, so it's fairly trivial to write a perl script that connects to the appropriate gmond and downloads the XML.My biggest concern at this point is getting the XML for several hundred hosts without blowing the memory on the system. After all, perl and XML::Simple are rather loose in their memory usage.
As a proof of concept, attached is a quick script I whipped together yesterday. The only problem I have with it right now is that it doesn't detect hosts that were down before gmond was started on the monitoring host. I can't blame the script for that, gmond just doesn't have the data to give me.
The scipt will generate output like: [EMAIL PROTECTED] % ./check_gmon.plOK: PDI Farm - 302 hosts (3 down) (cq135.pdi.com, cq251.pdi.com, cq282.pdi.com)
cq235.pdi.com (282.25 281.93 281.07) cq331.pdi.com (1023.32 163.58 971.62) cq387.pdi.com (305.41 305.11 304.16)and sets the return codes as appropriate for Nagios. (and yes, those are real load numbers from our cluster -- scary, isn't it? :) )
Again, this is only a proof of concept, it is not meant to be a complete monitoring tool. It monitors the entire cluster, not individual hosts, although it does return host info for down hosts and hosts with high loads. I'm going to see what I can do along those lines over the next few weeks, in my copious free time.... :)
-- Dan Rich <[EMAIL PROTECTED]> | http://www.employees.org/~drich/ | "Step up to red alert!" "Are you sure, sir? | It means changing the bulb in the sign..." | - Red Dwarf (BBC)
#!/usr/local/bin/perl
# $Id$
#
use strict;
use XML::Simple;
use Data::Dumper;
use Socket;
# Get the Nagios goodies
use lib "/usr/lib/nagios/plugins";
use utils qw($TIMEOUT %ERRORS &print_revision &support);
use vars qw($PROGNAME);
use Getopt::Long;
use vars qw($opt_V $opt_h $verbose $opt_w $opt_c $opt_H $opt_p);
$PROGNAME = "check_gmon";
sub print_help ();
sub print_usage ();
$ENV{'PATH'}='';
$ENV{'BASH_ENV'}='';
$ENV{'ENV'}='';
Getopt::Long::Configure('bundling');
GetOptions
("V" => \$opt_V, "version" => \$opt_V,
"h" => \$opt_h, "help" => \$opt_h,
"v" => \$verbose, "verbose" => \$verbose,
"p=s" => \$opt_p, "port=s" => \$opt_p,
"w=s" => \$opt_w, "warning=s" => \$opt_w,
"c=s" => \$opt_c, "critical=s" => \$opt_c,
"H=s" => \$opt_H, "hostname=s" => \$opt_H);
if ($opt_V) {
print_revision($PROGNAME,'$Revision: 1.4 $'); #'
exit $ERRORS{'OK'};
}
if ($opt_h) {
print_help();
exit $ERRORS{'OK'};
}
#$opt_H = shift unless ($opt_H);
#print_usage() unless ($opt_H);
my $host = $1 if ($opt_H =~
m/^([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+|[a-zA-Z][-a-zA-Z0]+(\.[a-zA-Z][-a-zA-Z0]+)*)$/);
#print_usage() unless ($host);
# Set reasonable defaults
my $critical;
my $warning;
if ($opt_H) {
($opt_c) || ($opt_c = shift) || ($opt_c = 8);
$critical = $1 if ($opt_c =~ /([0-9]+)/);
($opt_w) || ($opt_w = shift) || ($opt_w = 4);
$warning = $1 if ($opt_w =~ /([0-9]+)/);
} else {
if ($opt_c =~ /(\d+)%/) {$opt_c = $1 / 100;}
($opt_c) || ($opt_c = shift) || ($opt_c = 0.25);
$critical = $1 if ($opt_c =~ /([.0-9]+)/);
if ($opt_w =~ /(\d+)%/) {$opt_w = $1 / 100;}
($opt_w) || ($opt_w = shift) || ($opt_w = 0.10);
$warning = $1 if ($opt_w =~ /([.0-9]+)/);
}
# Parse ganglia config file
# Open socket to gmond
my $xml_port = $opt_p ? $opt_p : 8649;
my $proto = getprotobyname('tcp');
socket(GMOND, PF_INET, SOCK_STREAM, $proto);
my $sin = sockaddr_in($xml_port, inet_aton("127.0.0.1"));
connect(GMOND,$sin);
# Parse XML with XML::Simple
my $xs = new XML::Simple();
my $ref = $xs->XMLin(\*GMOND);
my $totalhosts;
my @downhosts = ();
my %highload;
foreach my $host ( @{$ref->{'CLUSTER'}->{'HOST'}} ) {
$totalhosts++;
if ($host->{'REPORTED'} < (time - 6000)) {
push(@downhosts,$host->{'NAME'});
} else {
# Get load averages
my($load_one,$load_five,$load_fifteen) = (undef,undef,undef);
foreach my $metric ( @{$host->{'METRIC'}} ) {
$load_one = $metric->{'VAL'} if ($metric->{'NAME'} eq "load_one");
$load_five = $metric->{'VAL'} if ($metric->{'NAME'} eq "load_five");
$load_fifteen = $metric->{'VAL'} if ($metric->{'NAME'} eq
"load_fifteen");
}
$highload{$host->{'NAME'}} = "($load_one $load_five $load_fifteen)"
if ($load_five > 5);
}
}
#print Dumper($ref);
if ($opt_H) {
} else {
my $result = 'OK';
if ($critical < 1) {
if ($#downhosts >= 0) {
if (($#downhosts + 1) / $totalhosts >= $critical) {
$result = 'CRITICAL';
} elsif (($#downhosts + 1) / $totalhosts >= $warning) {
$result = 'WARNING';
}
}
}
print "$result: ". $ref->{'CLUSTER'}->{'NAME'};
print " - ". $totalhosts ." hosts (". ($#downhosts + 1) ." down) ";
print "(". join(", ", sort @downhosts) .")" if ($verbose);
print "\n";
foreach my $hostname ( sort keys %highload ) {
print "\t$hostname\t$highload{$hostname}\n";
}
exit $ERRORS{$result};
}
sub print_usage () {
print "Usage: $PROGNAME [ -H <host> ] [ -p port ] [-w <warn>] [-c
<crit>]\n";
}
sub print_help () {
print_revision($PROGNAME,'$Revision: 1.4 $');
print "Copyright (c) 2000 Jeffery Blank/Karl DeBisschop\n";
print "\n";
print_usage();
print "\n";
print "<warn> = System load or number of down hosts at which a warning
message will be generated.\n";
print "<crit> = System load or number of down hosts at which a critical
message will be generated.\n\n";
print "The numbers above may be specified as either integers, or
percentages for number of hosts. The host numbers only apply if no hostname is
specified\n\n";
support();
print "\n\n";
}
signature.asc
Description: OpenPGP digital signature
