I've added Andy's initial description to the wiki as a placeholder: http://cwiki.apache.org/confluence/display/ESME/JMX+REST+API
@Andy: You may want to submit an Apache iCLA then you could edit the page yourself. D. On Sat, Nov 21, 2009 at 10:36 AM, Markus Kohler <[email protected]> wrote: > Hi Andy, > Dynatrace rocks, and Introscope is kind of the "old defacto standard. They > both can get the more detailed stats you mentioned below, but can also get > simple KPI's such as GC information. SNMP might also be an option to expose > simple KPI's. > > All we need is a list of methods that we would want to measure. > Those tools are then able to measure cpu time, elapsed time, nr. of calls > etc. > I > f we are really serious about this I think we should go for Dynatrace first. > I haven't checked whether they have an evaluation license. If not maybe we > can arrange some agreement with them. I happen to know some key people > there, because I once did an evaluation. > > Regards, > Markus > > On Fri, Nov 20, 2009 at 9:33 PM, Andy the destroyer < > [email protected]> wrote: > >> I am not familiar with Inroscope or Dynatrace. I will check them out. As >> far >> as instrumentation points, I think we need to find the use cases and they >> will dicate the technology or tools used. Right now we are just collecting >> Stats and we can change logging levels. With some changes to how esme and >> lift is configured we could do things like injecting new db servers to a >> connection pool, new request analyzers etc. However that sort of >> instrumentation is in my opinion better accomplished through OSGi. >> >> In stats collection I looked at some profiling tools that can collect >> method >> call counts, durations, exceptions thrown etc. I didn't use any of these >> because they slow down the JVM. From the names it sounds like Dynatrace >> provides this sort of data. Ill look into it. >> >> -Andy >> >> On Fri, Nov 20, 2009 at 12:01 AM, Markus Kohler <[email protected] >> >wrote: >> >> > Hi all, >> > Most enterprise monitoring systems understand JMX . IIRC the issue was >> > always that the remote JMX interface was vendor specific. Not sure >> whether >> > that's still the case. >> > JMX is rather inefficent I think. Most enterprises use products such as >> > Wiley Introscope (defacto SAP standard) or Dynatrace. >> > >> > We should probably look what interfaces they offer. Usually the main >> issue >> > is to find the right instrumentation points in the code. >> > >> > >> > Regards, >> > Markus >> > >> > On Fri, Nov 20, 2009 at 5:51 AM, Richard Hirsch <[email protected] >> > >wrote: >> > >> > > Just had another idea. >> > > >> > > @Andy: Is the pure JMX interface gone or just not active? I think it >> > > would be a interesting idea to be able to turn the real JMX interface >> > > on and off. If you can use the JMX interface (for example, behind the >> > > firewall in an on-premise installation), you turn it on. If you are in >> > > the cloud and can't use traditional JMX or don't have a use a >> > > enterprise monitoring system that understands JMX (for example, in >> > > smaller organizations), you turn it off. >> > > >> > > That way we could have the best of both worlds. >> > > >> > > Or another idea would be to offer a small component that reads the >> > > REST API and creates JMX out of it. If companies want JMX instead of >> > > REST, they could install this component. This solution might have >> > > performance probs though.... >> > > >> > > D. >> > > >> > > On Fri, Nov 20, 2009 at 5:43 AM, Richard Hirsch <[email protected] >> > >> > > wrote: >> > > > Posted message about SolutionManager and REST on the SDN forum: >> > > > http://forums.sdn.sap.com/thread.jspa?threadID=1536884 >> > > > >> > > > D. >> > > > >> > > > On Fri, Nov 20, 2009 at 5:31 AM, Richard Hirsch < >> [email protected] >> > > >> > > wrote: >> > > >> Looks great. >> > > >> >> > > >>>>There is no security on the api. This obviously should not stay >> like >> > > this and should only be accessable with an administrator >>account. >> > > >> >> > > >> You are probably correct about the necessity of authorization added >> to >> > > >> the interface but this might just be necessary for those interfaces >> > > >> that write / change settings, What about using the lift >> authorization >> > > >> mechanism here to distinguish it from the ESME user base? >> > > >> >> > > >>>It is worth thinking about switching to a property api like Configgy >> > > which uses mutable data structures for properties. >> > > >> >> > > >> This is a good idea. First we must determine which properties might >> be >> > > >> useful to be to change on the fly. I once started a list of such >> > > >> properties in the wiki. I'll finish it and publish the link. >> > > >> >> > > >> I really like the fact that JMX data is available via REST. I >> started >> > > >> looking and found there is a a related JSR 262, Web Services >> Connector >> > > >> for JMX Agents where the author examined rest-based access >> > > >> ( >> > > >> > >> http://blogs.sun.com/jmxnetbeans/entry/restful_access_to_jmx_instrumentation >> > > ). >> > > >> The author has some interesting ideas (for example, creating an >> Ajax >> > > >> based application to follow the JMX changes), I also found a very >> old >> > > >> Google Code project that looked at this topic >> > > >> (http://code.google.com/p/polarrose-jmx-rest-bridge/) where the >> > author >> > > >> mentions using Cacti (Cacti is a complete network graphing solution: >> > > >> http://www.cacti.net/) to graph the material. Maybe we could try >> and >> > > >> use Cacti as well to graph our data. >> > > >> >> > > >> So you could probably create a very cool administration application >> > > >> for ESME (written in whatever language you want) that allows you to >> > > >> monitor everything in ESME. What I'd like to see is a very basic >> admin >> > > >> app (maybe separate from ESME). Creating more complicated admin apps >> > > >> would then be a business opportunity for someone. >> > > >> >> > > >> What we have to look at as well is how to deal with those >> enterprises >> > > >> that already have monitoring environments in place (for example, >> > > >> SolMan). I don't know if such applications can deal with REST. I'll >> > > >> do some research here. >> > > >> >> > > >> So, in my opinion, it looks v. cool. Send me a patch and I'll commit >> > > >> it. Once we get the foundation up and running, we should talk about >> > > >> what other measures we want to monitor. >> > > >> >> > > >> Thanks. >> > > >> >> > > >> D. >> > > >> >> > > >> On Fri, Nov 20, 2009 at 3:55 AM, Andy the destroyer >> > > >> <[email protected]> wrote: >> > > >>> FYI & RFC >> > > >>> >> > > >>> I have finished a REST api for generic jmx MBeans. It allows for >> > > listing >> > > >>> mbeans, getting mbeaninfo, reading and setting attributes and >> > invoking >> > > >>> operations with parameters that can be represented as Strings ( >> i.e. >> > > String, >> > > >>> Long etc. serialized classes don't count). >> > > >>> >> > > >>> Before I send to code to Richard I would like some feedback to see >> if >> > I >> > > am >> > > >>> missing something or should do something different. >> > > >>> >> > > >>> Here is the api and sample returns. Bad requests or errors will >> > return >> > > >>> BadResponse. >> > > >>> >> > > >>> GET /jmx/mbeans - returns a list of mbean names >> > > >>> GET /jmx/mbean/[mbean name]/info - returns MBeanInfo for specified >> > > mbean >> > > >>> GET /jmx/mbean/[mbean name]/attributes - returns attribute name and >> > > string >> > > >>> values >> > > >>> >> > > >>> POST /jmx/mbean/[mbean name]/invoke/[operation name] >> > > >>> <Params> >> > > >>> <Param>pone</Param> >> > > >>> <Param signature="java.lang.Long">123</Param> >> > > >>> </Params> >> > > >>> >> > > >>> POST /jmx/mbean/[mbean name/set - sets a single attribute >> > > >>> <Attribute name="attr" value="val" /> >> > > >>> >> > > >>> POST /jmx/mbean/[mbean name]/set - sets multiple attributes >> > > >>> <AttributeList> >> > > >>> <Attribute name="att1" value="val1" /> >> > > >>> <Attrubute name="att2" value="val2" /> >> > > >>> </AttributeList> >> > > >>> >> > > >>> == SAMPLES == >> > > >>> >> > > >>> GET /jmx/mbeans >> > > >>> >> > > >>> <esme_api operation="mbeans" success="true"> >> > > >>> <MBeans> >> > > >>> <MBean name="java.lang:type=Memory"/> >> > > >>> <MBean name="java.lang:type=GarbageCollector,name=Copy"/> >> > > >>> <MBean name="java.lang:type=MemoryPool,name=Code Cache"/> >> > > >>> <MBean name="java.lang:type=Runtime"/> >> > > >>> <MBean name="java.lang:type=ClassLoading"/> >> > > >>> <MBean name="java.lang:type=MemoryPool,name=Perm Gen [shared-rw]"/> >> > > >>> <MBean name="java.lang:type=Threading"/> >> > > >>> <MBean name="java.util.logging:type=Logging"/> >> > > >>> <MBean name="java.lang:type=MemoryPool,name=Perm Gen [shared-ro]"/> >> > > >>> <MBean name="java.lang:type=Compilation"/> >> > > >>> <MBean name="java.lang:type=MemoryPool,name=Eden Space"/> >> > > >>> <MBean name="org.apache.esme.stats:type=Stats"/> >> > > >>> <MBean name="StatsAgent:name=htmlAdaptor,port=9092"/> >> > > >>> <MBean name="com.sun.management:type=HotSpotDiagnostic"/> >> > > >>> <MBean >> name="java.lang:type=GarbageCollector,name=MarkSweepCompact"/> >> > > >>> <MBean name="java.lang:type=MemoryPool,name=Survivor Space"/> >> > > >>> <MBean name="java.lang:type=MemoryPool,name=Tenured Gen"/> >> > > >>> <MBean name="java.lang:type=MemoryPool,name=Perm Gen"/> >> > > >>> <MBean name="java.lang:type=OperatingSystem"/> >> > > >>> <MBean name="JMImplementation:type=MBeanServerDelegate"/> >> > > >>> <MBean name="java.lang:type=MemoryManager,name=CodeCacheManager"/> >> > > >>> </MBeans> >> > > >>> </esme_api> >> > > >>> >> > > >>> GET /jmx/mbean/org.apache.esme.stats:type=Stats/attributes >> > > >>> <esme_api operation="mbean" success="true"> >> > > >>> <MBeanAttributes name="org.apache.esme.stats:type=Stats"> >> > > >>> <MBeanAttribute value="1" name="counter_userCount"/> >> > > >>> <MBeanAttribute value="1" name="counter_liftSessions"/> >> > > >>> <MBeanAttribute value="1.0" name="gauge_users"/> >> > > >>> <MBeanAttribute value="1.0" name="gauge_listener"/> >> > > >>> </MBeanAttributes> >> > > >>> </esme_api> >> > > >>> >> > > >>> GET /jmx/mbean/org.apache.esme.stats:type=Stats/info >> > > >>> <esme_api operation="mbean" success="true"> >> > > >>> <MBeanInfo name="org.apache.esme.stats:type=Stats"> >> > > >>> <MBeanAttributes> >> > > >>> <MBeanAttributeInfo isIs="false" type="java.lang.Long" >> > > description="counter" >> > > >>> isReadable="true" name="counter_userCount" isWritable="false"/> >> > > >>> <MBeanAttributeInfo isIs="false" type="java.lang.Long" >> > > description="counter" >> > > >>> isReadable="true" name="counter_liftSessions" isWritable="false"/> >> > > >>> <MBeanAttributeInfo isIs="false" type="java.lang.Long" >> > > description="gauge" >> > > >>> isReadable="true" name="gauge_users" isWritable="false"/> >> > > >>> <MBeanAttributeInfo isIs="false" type="java.lang.Long" >> > > description="gauge" >> > > >>> isReadable="true" name="gauge_listener" isWritable="false"/> >> > > >>> </MBeanAttributes> >> > > >>> <MBeanOperations> >> > > >>> <MBeanOperationInfo impact="ACTION" description="Remove all >> Counters, >> > > >>> Timers, Gauges and restart gathering timestamp." >> > > >>> returnType="java.lang.String" name="clear"> >> > > >>> <Signature> >> > > >>> </Signature> >> > > >>> </MBeanOperationInfo> >> > > >>> </MBeanOperations> >> > > >>> <MBeanNotifications/> >> > > >>> </MBeanInfo> >> > > >>> </esme_api> >> > > >>> >> > > >>> ================= >> > > >>> >> > > >>> There is no security on the api. This obviously should not stay >> like >> > > this >> > > >>> and should only be accessable with an administrator account. >> > > >>> >> > > >>> I tested it with the Stats MBean and the java.util.Logging MBean >> and >> > it >> > > is >> > > >>> working. I can access stats, reset them and change logging levels. >> > > >>> >> > > >>> It would be nice to be able to change runtime properties though JMX >> > > however >> > > >>> most of the Lift and ESME code uses vals and immutable data >> > structures >> > > that >> > > >>> are set when the app is originally loaded. The lift Props object >> used >> > > by >> > > >>> esme is totally immutable after it is constructed. It is worth >> > thinking >> > > >>> about switching to a property api like Configgy which uses mutable >> > data >> > > >>> structures for properties. >> > > >>> >> > > >>> We could change in real time things like resent & links period, >> > refresh >> > > >>> intervals, analyzer settings like long query time, db connection >> pool >> > > sizes >> > > >>> etc. >> > > >>> >> > > >>> I didn't enable sending serialized objects through the REST api >> > because >> > > it >> > > >>> seems like more trouble than it's worth. Besides that sort of JMX >> > > >>> functionality is only useful between servers and this api is really >> > > just for >> > > >>> the cloud environment like stax which doesn't allow you to bind to >> > > ports. >> > > >>> >> > > >>> I welcome any comments or suggestions of Stats or JMX features to >> add >> > > before >> > > >>> I send the code to Richard. Should invalid requests return specific >> > > error >> > > >>> messages or just BadResponse? >> > > >>> >> > > >>> Thanks, >> > > >>> Andy >> > > >>> >> > > >> >> > > > >> > > >> > >> >
