One thing to note about the solution that we created with mondemand, it
requires no changes to cassandra, and it works on any long running java
process with jmx. We basically have a few pieces
1. a command line JMX client which does 2 things
a. dumps a config file with all available jmx stats
b. when given that config file, attaches via jmx and every so often
dumps those statistics to either stdout, or via mondemand to the
network
2. a listener which captures mondemand packets, and writes rrd files
3. a web ui (which is basically drraw configured slightly differently),
which allows you to create graphs, composite graphs, templates and
dashboards.
Since the mondemand C library also contains a tool for sending statistics
we also can forward other information to this same box and graph it.
The mondemand java library is also embeddable, which is how we use it
for our other java based systems. However as I said it's not necessary
to embed to work if you don't mind another java process running to monitor
your first process.
We are currently working to get the mondemand-from-jmx library and command
line tool up with the rest of the mondemand framework on sourceforge.
-Anthony
On Tue, May 04, 2010 at 09:05:05PM +0300, Ran Tavory wrote:
> IMO there's a good case for both external monitoring tools and per-host
> minimalistic interface but I see Eric's point that every piece of code will
> require its maintenance. A cluster monitoring tool is definitely required.
> An embedded one has two nice properties:
> 1. It works out of the box
> 2. If I want to know NOW what's going on on the host, I don't have to poll
> like crazy (or wait 5 minutes for the next sample), I can just open the
> browser and see.
>
> I'll see if I can get something simple running and ping back, but I can't
> commit on a timeline.
> Is adding jetty acceptable? Any other preferences?
>
>
> On Tue, May 4, 2010 at 8:49 PM, Anthony Molinaro <
> antho...@alumni.caltech.edu> wrote:
>
> > And just to show you what the dashboards look like here's a couple
> > of screen shots
> >
> > Jconsole like page of jvm stats
> > http://herbie.ddv.com/~anthonym/mondemand-2.png
> >
> > Cassandra specific memtable stats
> > http://herbie.ddv.com/~anthonym/mondemand-1.png
> >
> > -Anthony
> >
> > On Tue, May 04, 2010 at 10:03:52AM -0700, Michael Lum wrote:
> > > On 5/4/2010 7:21 AM, Eric Evans wrote:
> > > >On Tue, 2010-05-04 at 08:41 +0300, Ran Tavory wrote:
> > > >>How about the following compromise:
> > > >>Add a simple web server to each node with only one simple servlet that
> > > >>simply spits out all JMX stats on one page. Not fancy, no graphs,
> > > >>simply the same values you can get from jconsole, but on a web page.
> > > >>To me it seems like a fair tradeoff b/w maintenance and easier out of
> > > >>the box management. Shooting up jconsole for each server is
> > > >>cumbersome, at least in the environment I work in (firewalls, high
> > > >>latency etc) so a web interface can be nice.
> > > >
> > > >It still seems superfluous to me, but I'd be open to something
> > > >fire-and-forget (i.e. wouldn't need updating each time something new was
> > > >added).
> > >
> > > This is how we monitor our Cassandra clusters. Each Cassandra node runs
> > > a process that polls the JMX stats and then fires off events to a set of
> > > configured management nodes using either UDP or multicast, depending on
> > > the network. New Cassandra nodes in the same cluster and datacenter
> > > have the same config (and are configured centrally anyways), and the
> > > management nodes automatically add new nodes based on the events they
> > > receive, so all the graphs, dashboards, monitors, and downstream tools
> > > pick all of this up without needing a change. This way we don't need to
> > > fire up jconsole for hundreds of nodes and can do other interesting
> > > cluster-wide aggregations. Also, we don't have to remember to setup
> > > monitoring when the cluster grows.
> > >
> > > All the tools used are open source, and I'd be happy to share more
> > > detail if there is interest.
> >
> > --
> > ------------------------------------------------------------------------
> > Anthony Molinaro <antho...@alumni.caltech.edu>
> >
--
------------------------------------------------------------------------
Anthony Molinaro <antho...@alumni.caltech.edu>
