Hi, I did HBASE-2219, it was a pretty bad hack, since it prevented even doing a uber-trivial RPC change without ruining background compatibility, and in my micro-benchmarks, it wasnt even faster to boot.
I was talking to Kannan earlier on IRC, and he was noting about multi-put vs put... I noted that nearly all Puts, even of Size=1 go through the multi-put code path now, and that seemed to be part of the problem as well. -ryan On Thu, Jul 29, 2010 at 3:50 PM, Gary Helmling <[email protected]> wrote: > Kannan, > > It looks like that code went away as part of HBASE-2219, which removed the > mapping of rpc method names to codes that was previously used in > serialization. > > We could do a similar initialization of the JMX MBean by just introspecting > the RPC interfaces though (HMasterInterface, HMasterRegionInterface, > HRegionInterface). Want to open a JIRA? > > Gary > > > On Thu, Jul 29, 2010 at 3:16 PM, Kannan Muthukkaruppan > <[email protected]>wrote: > >> Pushed the hadoop-metrics.properties changes, and it seems better but not >> quite. [It doesn't look like all the Region Servers are reporting similar >> set of counters (on some I am seeing "multiput" stuff, although we aren' t), >> lot of the average times are close to 0.] >> >> So next I looked into the HBASE-2146 patch, and that doesn't appear to be >> in trunk although at some point it was in trunk earlier. >> >> >> http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java?p2=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&p1=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&r1=901402&r2=901401&view=diff&pathrev=901402 >> >> Will try applying the patch locally to see if that helps. >> >> Regards, >> Kannan >> -----Original Message----- >> From: Kannan Muthukkaruppan [mailto:[email protected]] >> Sent: Thursday, July 29, 2010 2:41 PM >> To: [email protected] >> Subject: RE: RPC statistics from region servers... >> >> Gary, >> >> With some recent restructuring of our conf/* and cluster deploy scripts, we >> did lose our hadoop-metrics.properties changes, and reverted back to the >> default one. That could very well report the issue I ran into. I'll report >> back when I try it out. >> >> Thanks again for your help on this. >> >> Q: Why don't we make the suggested settings in >> http://hbase.apache.org/docs/current/metrics.html be part of the default >> conf/hadoop-metrics.properties? >> >> Regards, >> Kannan >> -----Original Message----- >> From: Gary Helmling [mailto:[email protected]] >> Sent: Thursday, July 29, 2010 1:08 AM >> To: [email protected] >> Subject: Re: RPC statistics from region servers... >> >> Hmm, this sounds like HBASE-2146. That patch had added code to initialize >> the RPC metrics MBean with a know list of RPC method names maintained in >> HBaseRPC.Invocation. That way all per-method metrics would show up on >> server startup. I believe that that method name mapping was later removed >> though (it can make the RPC serialization brittle, complicating things like >> rolling restarts), so maybe the patch that was applied to trunk was later >> removed? >> >> The MBean attributes for individual RPC methods _should_ be there after >> there has been activity involving those requests against the cluster. If >> you start jconsole prior to the cluster activity however, I don't think the >> MBean definition will update, so you'll need to restart jconsole for them >> to >> show up. >> >> In addition, you should make sure you're running with a metrics context >> with >> an updater configured in >> conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext, >> GangliaContext, etc). And configure rpc.period with the update interval. >> These metrics don't show up as live counters in JMX, they're only pushed >> out >> to JMX during the periodic updates. >> >> If you've already configured all of those conditions and the metrics >> _still_ >> aren't showing up, then let me know and I'll try to do some testing on >> trunk >> tomorrow. >> >> --gh >> >> >> On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan >> <[email protected]>wrote: >> >> > > they don't exist until used? >> > >> > I was doing a whole bunch of puts & gets against the cluster during that >> > time. That's what you mean, correct? >> > >> > Regards, >> > Kannan >> > -----Original Message----- >> > From: Ryan Rawson [mailto:[email protected]] >> > Sent: Wednesday, July 28, 2010 8:21 PM >> > To: [email protected] >> > Cc: [email protected] >> > Subject: Re: RPC statistics from region servers... >> > >> > I think those are dynamic metrics... they don't exist until used? >> > >> > On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <[email protected]> >> > wrote: >> > > I was using jconsole to check the stats published by region servers.. >> > > >> > > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see: >> > > >> > > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime) >> > > and, >> > > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime). >> > > >> > > I no longer see attributes like: >> > > "incrementColumnValueNumOps " >> > > "incrementColumnValueAvgTime " >> > > "putNumOps " >> > > "putAvgTime " >> > > "getNumOps " >> > > "getAvgTime " >> > > "deleteNumOps " >> > > "deleteAvgTime " >> > > "openScannerNumOps " >> > > "openScannerAvtTime " >> > > "nextNumOps " >> > > "nextAvgTime " >> > > "closeNumOps " >> > > "closeAvgTime " >> > > >> > > Anyone aware of what change happened in this area? >> > > >> > > regards, >> > > Kannan >> > > >> > > >> > >> >
