I do seem to be getting data in multiPut* and get* now from all the Region Servers. So things look good now.
Gary: > We could do a similar initialization of the JMX MBean by just introspecting > the RPC interfaces though (HMasterInterface, HMasterRegionInterface, > HRegionInterface). What would be the net effect of this change compared to the state trunk is currently in? Could you clarify? Regards, Kannan -----Original Message----- From: Gary Helmling [mailto:[email protected]] Sent: Thursday, July 29, 2010 3:50 PM To: [email protected] Subject: Re: RPC statistics from region servers... Kannan, It looks like that code went away as part of HBASE-2219, which removed the mapping of rpc method names to codes that was previously used in serialization. We could do a similar initialization of the JMX MBean by just introspecting the RPC interfaces though (HMasterInterface, HMasterRegionInterface, HRegionInterface). Want to open a JIRA? Gary On Thu, Jul 29, 2010 at 3:16 PM, Kannan Muthukkaruppan <[email protected]>wrote: > Pushed the hadoop-metrics.properties changes, and it seems better but not > quite. [It doesn't look like all the Region Servers are reporting similar > set of counters (on some I am seeing "multiput" stuff, although we aren' t), > lot of the average times are close to 0.] > > So next I looked into the HBASE-2146 patch, and that doesn't appear to be > in trunk although at some point it was in trunk earlier. > > > http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java?p2=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&p1=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&r1=901402&r2=901401&view=diff&pathrev=901402 > > Will try applying the patch locally to see if that helps. > > Regards, > Kannan > -----Original Message----- > From: Kannan Muthukkaruppan [mailto:[email protected]] > Sent: Thursday, July 29, 2010 2:41 PM > To: [email protected] > Subject: RE: RPC statistics from region servers... > > Gary, > > With some recent restructuring of our conf/* and cluster deploy scripts, we > did lose our hadoop-metrics.properties changes, and reverted back to the > default one. That could very well report the issue I ran into. I'll report > back when I try it out. > > Thanks again for your help on this. > > Q: Why don't we make the suggested settings in > http://hbase.apache.org/docs/current/metrics.html be part of the default > conf/hadoop-metrics.properties? > > Regards, > Kannan > -----Original Message----- > From: Gary Helmling [mailto:[email protected]] > Sent: Thursday, July 29, 2010 1:08 AM > To: [email protected] > Subject: Re: RPC statistics from region servers... > > Hmm, this sounds like HBASE-2146. That patch had added code to initialize > the RPC metrics MBean with a know list of RPC method names maintained in > HBaseRPC.Invocation. That way all per-method metrics would show up on > server startup. I believe that that method name mapping was later removed > though (it can make the RPC serialization brittle, complicating things like > rolling restarts), so maybe the patch that was applied to trunk was later > removed? > > The MBean attributes for individual RPC methods _should_ be there after > there has been activity involving those requests against the cluster. If > you start jconsole prior to the cluster activity however, I don't think the > MBean definition will update, so you'll need to restart jconsole for them > to > show up. > > In addition, you should make sure you're running with a metrics context > with > an updater configured in > conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext, > GangliaContext, etc). And configure rpc.period with the update interval. > These metrics don't show up as live counters in JMX, they're only pushed > out > to JMX during the periodic updates. > > If you've already configured all of those conditions and the metrics > _still_ > aren't showing up, then let me know and I'll try to do some testing on > trunk > tomorrow. > > --gh > > > On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan > <[email protected]>wrote: > > > > they don't exist until used? > > > > I was doing a whole bunch of puts & gets against the cluster during that > > time. That's what you mean, correct? > > > > Regards, > > Kannan > > -----Original Message----- > > From: Ryan Rawson [mailto:[email protected]] > > Sent: Wednesday, July 28, 2010 8:21 PM > > To: [email protected] > > Cc: [email protected] > > Subject: Re: RPC statistics from region servers... > > > > I think those are dynamic metrics... they don't exist until used? > > > > On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <[email protected]> > > wrote: > > > I was using jconsole to check the stats published by region servers.. > > > > > > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see: > > > > > > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime) > > > and, > > > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime). > > > > > > I no longer see attributes like: > > > "incrementColumnValueNumOps " > > > "incrementColumnValueAvgTime " > > > "putNumOps " > > > "putAvgTime " > > > "getNumOps " > > > "getAvgTime " > > > "deleteNumOps " > > > "deleteAvgTime " > > > "openScannerNumOps " > > > "openScannerAvtTime " > > > "nextNumOps " > > > "nextAvgTime " > > > "closeNumOps " > > > "closeAvgTime " > > > > > > Anyone aware of what change happened in this area? > > > > > > regards, > > > Kannan > > > > > > > > >
