On Mon, Feb 23, 2009 at 12:42:30PM -0500, Ofer Inbar wrote:
> Johann Spies <[email protected]> wrote:
> > On our 21-node cluster the graphs stopped showing what is going on on
> > the nodes. The graphs for the head of the cluster (where gmetad is
> > running) is normal. Although there are no data shown in the graphs
> > ganglia seems to get data from the nodes because the the colour of the
> > nodes change when the load increases and the values for e.g. load
> > average are printed but not graphed.
>
> > Ganglia version 2.5.7-99 on OpenSuse 10.3.
>
> That's pretty ancient, so I'm not sure if I know where I'd look, or if
> all of my advice below applies. If you were running 3.0 or 3.1, I'd say:
2.5.7-99 was the version I could get as OpenSuse-package.
> However:
>
> 1. Have you checked your syslog on the gmetad node
> (/var/log/messages or wherever it goes on your system) for messages
> from gmetad?
Nothing that indicates a problem found in /var/log/messaages.
>
> 2. ... and the syslog on all nodes for messages from gmond?
The last errors there was from November last year when I was setting
up the system for the first time. Since then it worked without a
problem for at least two months.
>
> 3. Can you get the XML from port 8649?
> Does it include all the nodes & metrics?
That seems OK.
>
> 4. Where are your .rrd files stored?
/var/lib/ganglia
> Are they all there?
Yes
> Do they seem to be getting new data?
The files for the nodes seems to be outdated while the files in
__SummaryInfo__ reflect the current time in the timestamps.
I did the following: stopped gmetad on the head, moved the directory
containing the rrd-files to another name, created an empty directory
and restarted gmetad. Only the rrd's for the head node were
created. I then restarted all the gmond-processes on the nodes and the
rest of the rrd-files were created.
As I don't want to lose all the historical data, I moved the original
directory back and restarted the gmond-processes again. Now the
graphing seems to work.
> Is that disk full?
No.
Thank you for your reply. Your questions have helped me to debug the
problem.
Regards.
Johann
--
Johann Spies Telefoon: 021-808 4036
Informasietegnologie, Universiteit van Stellenbosch
"I will lift up mine eyes unto the hills, from whence
cometh my help. My help cometh from the LORD, which
made heaven and earth."
Psalms 121:1,2
------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
