[ 
https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117621#comment-13117621
 ] 

jirapos...@reviews.apache.org commented on HBASE-4145:
------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1674/#review2175
-----------------------------------------------------------


Only one minor comment left, see below.
If test suite passes, I would vote +1


http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java
<https://reviews.apache.org/r/1674/#comment5068>

    VersionedWritable from hadoop left something to be desired.
    Such discussion came up during HBASE-2195.
    We can address this elsewhere.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java
<https://reviews.apache.org/r/1674/#comment5069>

    I meant that this counter can be named REGIONS_SCANNED because its value 
may be lower than the total number of regions in the table(s).


- Ted


On 2011-09-29 21:00:18, Ming Ma wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1674/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-09-29 21:00:18)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  1. Collect client-side scan related metrics during scan operation. It is 
turned off by default.
bq.  2. TableInputFormat enables metrics collection on scan and pass the data 
to mapreduce framework. It only works with new mapreduce APIs that allow 
TableInputFormat to get access to mapreduce Counter.
bq.  3. Clean up some minor issues in tableInputFormat as well as test code.
bq.  
bq.  
bq.  This addresses bug hbase-4145.
bq.      https://issues.apache.org/jira/browse/hbase-4145
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1176942 
bq.    
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
 1176942 
bq.    
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java
 1176942 
bq.    
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
 1176942 
bq.    
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java
 PRE-CREATION 
bq.    
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
 1176942 
bq.    
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java
 1176942 
bq.    
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
 1176942 
bq.    
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1176942 
bq.    
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java
 1176942 
bq.  
bq.  Diff: https://reviews.apache.org/r/1674/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  1. Verified on a small cluster.
bq.  2. Existing unit tests.
bq.  3. Added new tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Ming
bq.  
bq.


                
> Provide metrics for hbase client
> --------------------------------
>
>                 Key: HBASE-4145
>                 URL: https://issues.apache.org/jira/browse/HBASE-4145
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Ming Ma
>         Attachments: HBaseClientSideMetrics.jpg
>
>
> Sometimes it is useful to get some metrics from hbase client point of view. 
> This will help understand the metrics for scan/TableInputFormat map job 
> scenario.
> What to capture, for example, for each ResultScanner object,
> 1. The number of RPC calls to RSs.
> 2. The delta time between consecutive RPC calls in the current serialized 
> scan implementation.
> 3. The number of RPC retry to RSs.
> 4. The number of NotServingRegionException got.
> 5. The number of remote RPC calls. This excludes those call that hbase client 
> calls the RS on the same machine.
> 6. The number of regions accessed.
> How to capture
> 1. Metrics framework works for a fixed number of metrics. It doesn't fit this 
> scenario.
> 2. Use some TBD solution in HBase to capture such dynamic metrics. If we 
> assume there is a solution in HBase that HBase client can use to log such 
> kind of metrics, TableInputFormat can pass in mapreduce task ID as 
> application scan ID to HBase client as small addition to existing scan API; 
> and HBase client can log metrics accordingly with such ID. That will allow 
> query, analysis later on the metrics data for specific map reduce job.
> 3. Expose via MapReduce counter. It lacks certain features, for example, 
> there is no good way to access the metrics on per map instance; the MapReduce 
> framework only performs sum on the counter values so it is tricky to find the 
> max of certain metrics in all mapper instances. However, it might be good 
> enough for now. With this approach, the metrics value will be available via 
> MapReduce counter.
> a) Have ResultScanner return a new ResultScannerMetrics interface.
> b) TableInputFormat will access data from ResultScannerMetrics and populate 
> MapReduce counters accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Reply via email to