On Monday, February 25, 2013, Felix Meschberger wrote:
> Hi,
>
> Am 22.02.2013 um 23:59 schrieb Ian Boston:
>
> > Hi,
> > Whilst writing the MBeans in the event bundle I started thinking about
> > monitoring inside Sling. IMHO there are not enough to really know what
> > a instance under load is doing. Much though I like JMX it comes with
> > implementation and runtime overhead which I don't much like.
> >
> > Runtime:
> > * Running with JMX enabled doesn't add any overhead, but once a client
> > is connected there is some (some reports upto 3% of resources).
>
> I agree. But a client connecting (actually it is not the connection per
> se, it is the requests executed by the client, which bring the load) may
> load the system, which is ok, because the client is investigating.
>
>
> > * You have to remember to enable it, and most of the time JVMs are not
> > enabled. By the time you really need it, its often too late.
> > * JMX is not restful.
>
> I don't think that this is enough of a reason to reinvent the wheel given
> the spread of JMX support in tools.
>
> >
> > Implementation
> > * MBeans are not that hard to implement with the OSGi Whiteboard, but
> > they have to be implemented.
>
> Right. Everyhting that has to be queried has to be implemented. The actual
> problem I see, is that a separate API is required, which in turn is good
> programming style anyway when doing services. Yet this API does not need to
> be exported because the JMX Whiteboard gets to it through the MBean service
> class (IIRC).
>
> >
> > Alternatives.
> > In Jackarabbit there is/was a statistics class [1], which IIUC uses
> > counters and time series stored in a map. The service can then be
> > queries to extract the values by wrapping in an MBean or Servlet.
>
> The only advantage over direct JMX I see, is the Servlet wrapping
> (probably the Web Console in our case ?). How common is that use case ?
I think it's quite common when running more than a few instances to want to
be able to quickly gather read only stats from all the instances. Ie a json
feed per server. The easiest way of doing that is to have a single end
point that delivers a bundle of counters with a time stamp. If that's done
then all of the time series and trending can be performed by the monitoring
tool that can be much more efficient at the task. ( ie Munin, Reconoiter,
Splunk etc)
The main problem with jmx in a multiple instance monitoring environment is
two fold.
It opens up all sorts of admin level operations.
The jmx protocol is rpc like in nature requiring many network rounds trips.
I am not saying it cant be done with jmx, its just not very efficient and
with more than a handfull of jvms has to be done on the instance in
question, side stepping the scaling issue.
As for the suggested service. The reason for the centralised map of
counters is so that the get operation can just serialise that map, rather
than having to invoke get attributes on many Mbeans. The other reason for
the centralised map of counters is so that areas that need to record usage,
can do so with single lines of code. It's quite close to the pattern of
RepositoryStatislticsImpl in Jackrabbit except there an enum is used to
control the map and the objects are all time series.
I agree using jmx for monitoring of one or two jvms does make sense and it
would not be a good idea to reinvent that wheel. It would certainly be a
mistake to reinvent operations or notifications.
Ian
>
> And even then I would prefer a good DTO design over just a "counter map".
>
> The StatisticsUtil class is IMHO overkill.
>
> Regards
> Felix
>
> >
> > I think the approach could be generalised and extended so that
> > anything in the container could use the service to record metrics. The
> > api might look something like
> >
> > public interface Statistics {
> >
> > /**
> > * Increment a counter by 1
> > */
> > void increment(String counterName);
> >
> > /**
> > * Record a double value in a timeseries.
> > */
> > void record(String timeSeriesName, double value);
> >
> > /**
> > * Record a long value in a timeseries.
> > */
> > void record(String timeSeriesName, long value);
> >
> > }
> >
> > and (so that any reference can be optional on a service
> > implementation, the final is a hint to hotspot to inline)
> >
> > public final class StatisticsUtils {
> >
> > private StatisticsUtils() {
> > }
> >
> > public static void increment(Statistics statistics, String counterName)
> {
> > if ( statistics != null ) {
> > statistics.increment(counterName);
> > }
> > }
> >
> > ... etc for the other methods ..
> > }
> >
> >
> >
> >
> > The service would need to deal with all the implementation details
> > (including concurrency and speed). The service implementation would
> > also come with a servlet endpoint (under /system/*) and/or single JMX
> > MBean.
> >
> > Anything that wanted to record stats would then bind to the service
> > and use it. I think this would avoid the issues mentioned above with
> > wide scale MBean usage.
> >
> > WDYT?
> >
> > (apologies for the noise if this already exists, and if so, please
> > treat it as a question: where and how do we record stats?)
> >
> > Ian
> >
> >
> >
> >
> > 1 http://wiki.apache.org/jackrabbit/Statistics
>
>
> --
> Felix Meschberger | Principal Scientist | Adobe
>
>
>
>
>
>
>
>
