[jira] [Resolved] (HBASE-5568) Multi concurrent flushcache() for one region could cause data loss
[ https://issues.apache.org/jira/browse/HBASE-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-5568. --- Resolution: Fixed Hadoop Flags: Reviewed Integrated to 0.92 Multi concurrent flushcache() for one region could cause data loss -- Key: HBASE-5568 URL: https://issues.apache.org/jira/browse/HBASE-5568 Project: HBase Issue Type: Bug Components: regionserver Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0 Attachments: HBASE-5568-90.patch, HBASE-5568-92v2.patch, HBASE-5568.patch, HBASE-5568.patch, HBASE-5568v2.patch We could call HRegion#flushcache() concurrently now through HRegionServer#splitRegion or HRegionServer#flushRegion by HBaseAdmin. However, we find if HRegion#internalFlushcache() is called concurrently by multi thread, HRegion.memstoreSize will be calculated wrong. At the end of HRegion#internalFlushcache(), we will do this.addAndGetGlobalMemstoreSize(-flushsize), but the flushsize may not the actual memsize which flushed to hdfs. It cause HRegion.memstoreSize is negative and prevent next flush if we close this region. Logs in RS for region e9d827913a056e696c39bc569ea3 2012-03-11 16:31:36,690 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 f99f., current region memstore size 128.0m 2012-03-11 16:31:37,999 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e a3f99f/cf1/8162481165586107427, entries=153106, sequenceid=619316544, memsize=59.6m, filesize=31.2m 2012-03-11 16:31:38,830 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 f99f., current region memstore size 134.8m 2012-03-11 16:31:39,458 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e a3f99f/cf2/3425971951499794221, entries=230183, sequenceid=619316544, memsize=68.5m, filesize=26.6m 2012-03-11 16:31:39,459 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~128.1m for region writetest1,,1331454657410.e9d827913a 056e696c39bc569ea3f99f. in 2769ms, sequenceid=619316544, compaction requested=false 2012-03-11 16:31:39,459 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 f99f., current region memstore size 6.8m 2012-03-11 16:31:39,529 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e a3f99f/cf1/1811012969998104626, entries=8002, sequenceid=619332759, memsize=3.1m, filesize=1.6m 2012-03-11 16:31:39,640 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e a3f99f/cf2/770333473623552048, entries=12231, sequenceid=619332759, memsize=3.6m, filesize=1.4m 2012-03-11 16:31:39,641 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~134.8m for region writetest1,,1331454657410.e9d827913a 056e696c39bc569ea3f99f. in 811ms, sequenceid=619332759, compaction requested=true 2012-03-11 16:31:39,707 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e a3f99f/cf1/5656568849587368557, entries=119, sequenceid=619332979, memsize=47.4k, filesize=25.6k 2012-03-11 16:31:39,775 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e a3f99f/cf2/794343845650987521, entries=157, sequenceid=619332979, memsize=47.8k, filesize=19.3k 2012-03-11 16:31:39,777 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~6.8m for region writetest1,,1331454657410.e9d827913a05 6e696c39bc569ea3f99f. in 318ms, sequenceid=619332979, compaction requested=true -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3848) request is always zero in WebUI for region server
[ https://issues.apache.org/jira/browse/HBASE-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-3848. --- Resolution: Fixed The remaining work would be completed by HBASE-4977 request is always zero in WebUI for region server - Key: HBASE-3848 URL: https://issues.apache.org/jira/browse/HBASE-3848 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.2 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Attachments: RegionServer_90PatchV2.patch, RegionseverMetric_TrunkPathV2.patch request is always zero in WebUI for region server Metrics request=0.0, regions=36, stores=36, storefiles=148, storefileIndexSize=29, memstoreSize=253, compactionQueueSize=24, flushQueueSize=0, usedHeap=655, maxHeap=8175, blockCacheSize=14230920, blockCacheFree=1700269560, blockCacheCount=21, blockCacheHitCount=2887, blockCacheMissCount=204829, blockCacheEvictedCount=0, blockCacheHitRatio=1, blockCacheHitCachingRatio=99 requests is not zero in WebUI for Hmaster requests=15000, regions=35, usedHeap=513, maxHeap=8175 Is there any different for these metrics? How do I use it? Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4856) Upgrade zookeeper to 3.4.0 release
[ https://issues.apache.org/jira/browse/HBASE-4856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4856. --- Resolution: Fixed Upgrade zookeeper to 3.4.0 release -- Key: HBASE-4856 URL: https://issues.apache.org/jira/browse/HBASE-4856 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4856.txt Zookeeper 3.4.0 has been released. We should upgade. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4793) HBase shell still using deprecated methods removed in HBASE-4436
[
https://issues.apache.org/jira/browse/HBASE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-4793.
---
Resolution: Fixed
Assignee: Gary Helmling
Hadoop Flags: Reviewed
HBase shell still using deprecated methods removed in HBASE-4436
Key: HBASE-4793
URL: https://issues.apache.org/jira/browse/HBASE-4793
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 0.92.0, 0.94.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Critical
Fix For: 0.92.0
Attachments: 4793.txt
The patch applied in HBASE-4622 (subtask of HBASE-4436) to remove deprecated
methods seems to have missed some usage of those methods by the HBase shell.
At least src/main/ruby/hbase/admin.rb is still using some of the removed
methods, breaking some shell commands:
{noformat}
hbase(main):007:0 alter 'privatetable', { NAME = 'f1', VERSIONS = 2}
ERROR: wrong number of arguments (3 for 2)
Backtrace: /usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:344:in `alter'
org/jruby/RubyArray.java:1572:in `each'
/usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:317:in `alter'
/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:79:in `command'
/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:68:in
`format_simple_command'
/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:78:in `command'
/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in
`command_safe'
/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:74:in
`translate_hbase_exceptions'
/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in
`command_safe'
/usr/lib/hbase/bin/../bin/../lib/ruby/shell.rb:110:in `command'
(eval):2:in `alter'
{noformat}
This trace translates to the line:
{code}
@admin.modifyColumn(table_name, column_name, descriptor)
{code}
which is calling one of the removed methods.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4704) A JRuby script for identifying active master
[ https://issues.apache.org/jira/browse/HBASE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4704. --- Resolution: Fixed Hadoop Flags: Reviewed A JRuby script for identifying active master Key: HBASE-4704 URL: https://issues.apache.org/jira/browse/HBASE-4704 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Trivial Fix For: 0.94.0 Attachments: D117.1.patch, HBASE-4704.D117.2.patch, HBASE-4704.D117.3.patch This simple script reads the HBase master ZK node and outputs the hostname of the active master. This is needed so that operational scripts can decide where the primary master is running. I am also including a one-line hbase-jruby script so we can make our jruby scripts proper UNIX executables by including an #!/usr/bin/env hbase-jruby at the top. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4694) Some cleanup of log messages in RS and M
[ https://issues.apache.org/jira/browse/HBASE-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4694. --- Resolution: Fixed Test failures in TRUNK is down to 1. Some cleanup of log messages in RS and M Key: HBASE-4694 URL: https://issues.apache.org/jira/browse/HBASE-4694 Project: HBase Issue Type: Improvement Reporter: stack Assignee: stack Fix For: 0.92.0 Attachments: 4694.addendum, logging.txt I did a little edit of logging. We do way too much but am not going to do a big overhaul just yet. Here's a few small changes saving a few lines, some redundancy, and making others look like surrounding log lines. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter
[ https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4532. --- Resolution: Fixed Hadoop Flags: Reviewed Integrated addendum to TRUNK. Thanks Jonathan. Avoid top row seek by dedicated bloom filter for delete family bloom filter --- Key: HBASE-4532 URL: https://issues.apache.org/jira/browse/HBASE-4532 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, hbase-4532-89-fb.patch, hbase-4532-remove-system.out.println.patch The previous jira, HBASE-4469, is to avoid the top row seek operation if row-col bloom filter is enabled. This jira tries to avoid top row seek for all the cases by creating a dedicated bloom filter only for delete family The only subtle use case is when we are interested in the top row with empty column. For example, we are interested in row1/cf1:/1/put. So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family bloom filter will say there is NO delete family. Then it will avoid the top row seek and return a fake kv, which is the last kv for this row (createLastOnRowCol). In this way, we have already missed the real kv we are interested in. The solution for the above problem is to disable this optimization if we are trying to GET/SCAN a row with empty column. Evaluation from TestSeekOptimization: Previously: For bloom=NONE, compr=NONE total seeks without optimization: 2506, with optimization: 1714 (68.40%), savings: 31.60% For bloom=ROW, compr=NONE total seeks without optimization: 2506, with optimization: 1714 (68.40%), savings: 31.60% For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with optimization: 1458 (58.18%), savings: 41.82% For bloom=NONE, compr=GZ total seeks without optimization: 2506, with optimization: 1714 (68.40%), savings: 31.60% For bloom=ROW, compr=GZ total seeks without optimization: 2506, with optimization: 1714 (68.40%), savings: 31.60% For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with optimization: 1458 (58.18%), savings: 41.82% So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is enabled.[HBASE-4469] After this change: For bloom=NONE, compr=NONE total seeks without optimization: 2506, with optimization: 1458 (58.18%), savings: 41.82% For bloom=ROW, compr=NONE total seeks without optimization: 2506, with optimization: 1458 (58.18%), savings: 41.82% For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with optimization: 1458 (58.18%), savings: 41.82% For bloom=NONE, compr=GZ total seeks without optimization: 2506, with optimization: 1458 (58.18%), savings: 41.82% For bloom=ROW, compr=GZ total seeks without optimization: 2506, with optimization: 1458 (58.18%), savings: 41.82% For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with optimization: 1458 (58.18%), savings: 41.82% So we can get about 10% more seek savings for ALL kinds of bloom filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4651) ConcurrentModificationException might be thrown in TestHCM.testConnectionUniqueness
[
https://issues.apache.org/jira/browse/HBASE-4651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-4651.
---
Resolution: Fixed
Hadoop Flags: Reviewed
ConcurrentModificationException might be thrown in
TestHCM.testConnectionUniqueness
---
Key: HBASE-4651
URL: https://issues.apache.org/jira/browse/HBASE-4651
Project: HBase
Issue Type: Test
Affects Versions: 0.92.0
Reporter: gaojinchao
Assignee: gaojinchao
Priority: Minor
Fix For: 0.92.0
Attachments: HBASE-4651_Trunk.patch
See
https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2357/testReport/junit/org.apache.hadoop.hbase.client/TestHCM/testConnectionUniqueness
363 EntryK,V nextEntry() {
364 if (modCount != expectedModCount)
365 throw new
ConcurrentModificationExceptionhttp://kickjava.com/src/java/util/ConcurrentModificationException.java.htm
[image:
JavaDoc] http://kickjava.com/2487.htm();
Read more:
http://kickjava.com/src/java/util/LinkedHashMap.java.htm#ixzz1bbCC0gaT
HCM uses proper synchronization when accessing HBASE_INSTANCES.
Looking at TestHCM.getValidKeyCount(), it puts values of HBASE_INSTANCES in a
Set and returns the size of the Set.
However, post HBASE-3777, the values (HConnectionImplementation's) in
HBASE_INSTANCES would be unique.
TestHCM.getValidKeyCount() can be removed from the test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4203) While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts
[ https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4203. --- Resolution: Fixed This was covered in HBASE-4015 While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts Key: HBASE-4203 URL: https://issues.apache.org/jira/browse/HBASE-4203 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor 1. Start Master and 2 RS. 2. If any exception happens while opening the META region the state in znode will be OPENING. 3. If at this point the master restarts then the master will start processing the regions in RIT. 4. If the znode is found to be in OPENING then master waits for timeout monitor to deduct and then call opening. 5. If default timeout monitor is configured(180 sec/30 min) then it will take 30 mins to open the META region itself. Soln: Better not to wait for the Timeout monitor period to open catalog tables on Master restart -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4474) Use variable length format to store the memstoreTS
[ https://issues.apache.org/jira/browse/HBASE-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4474. --- Resolution: Duplicate This has been covered in the latest patch for HBASE-2856 Use variable length format to store the memstoreTS -- Key: HBASE-4474 URL: https://issues.apache.org/jira/browse/HBASE-4474 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Fix For: 0.92.0 HBASE-4344 introduced memstoreTS for KeyValues. The following suggestion was from Kannan: We should consider using variable length format to store the memstoreTS on disk. Also, at the start of the flush, we can probably prune most of these timestamps to 0 since only the ones that are higher than the current read point for all active scanners need to be maintained at the fine grain level. So like often times, for a majority of the KVs, we might be able to just write a 0. And hence, storing in varying width format would be an even bigger win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4496) HFile V2 does not honor setCacheBlocks when scanning.
[ https://issues.apache.org/jira/browse/HBASE-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4496. --- Resolution: Fixed Hadoop Flags: Reviewed HFile V2 does not honor setCacheBlocks when scanning. - Key: HBASE-4496 URL: https://issues.apache.org/jira/browse/HBASE-4496 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0, 0.94.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.92.0, 0.94.0 Attachments: 4496.final, 4496.txt While testing the LRU cache during the scanning I noticed quite some churn in the cache even when Scan.cacheBlocks is set to false. After debugging this, I found that HFile V2 always caches blocks in the LRU cache regardless of the cacheBlocks setting. Here's a trace (from Eclipse) showing the problem: HFileReaderV2.readBlock(long, int, boolean, boolean, boolean) line: 279 HFileReaderV2.readBlockData(long, long, int, boolean) line: 219 HFileBlockIndex$BlockIndexReader.seekToDataBlock(byte[], int, int, HFileBlock) line: 191 HFileReaderV2$ScannerV2.seekTo(byte[], int, int, boolean) line: 502 HFileReaderV2$ScannerV2.reseekTo(byte[], int, int) line: 539 StoreFileScanner.reseekAtOrAfter(HFileScanner, KeyValue) line: 151 StoreFileScanner.reseek(KeyValue) line: 110 KeyValueHeap.reseek(KeyValue) line: 255 StoreScanner.reseek(KeyValue) line: 409 StoreScanner.next(ListKeyValue, int) line: 304 KeyValueHeap.next(ListKeyValue, int) line: 114 KeyValueHeap.next(ListKeyValue) line: 143 HRegion$RegionScannerImpl.nextRow(byte[]) line: 2774 HRegion$RegionScannerImpl.nextInternal(int) line: 2722 HRegion$RegionScannerImpl.next(ListKeyValue, int) line: 2682 HRegion$RegionScannerImpl.next(ListKeyValue) line: 2699 HRegionServer.next(long, int) line: 2092 Every scanner.next causes a reseek, which eventually causes a call to HFileBlockIndex$BlockIndexReader.seekToDataBlock(...) at which point the cacheBlocks information is lost. HFileReaderV2.readBlockData calls HFileReaderV2.readBlock with cacheBlocks set unconditionally to true. The fix is not immediately clear, unless we want to pass cacheBlocks to HFileBlockIndex$BlockIndexReader.seekToDataBlock and then on to HFileBlock.BasicReader.readBlockData and all its implementers, which is ugly as readBlockData should not care about caching. Avoiding caching during scans is somewhat important for us. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4412) No need to retry scan operation on the same server in case of RegionServerStoppedException
[
https://issues.apache.org/jira/browse/HBASE-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-4412.
---
Resolution: Fixed
Hadoop Flags: Reviewed
No need to retry scan operation on the same server in case of
RegionServerStoppedException
--
Key: HBASE-4412
URL: https://issues.apache.org/jira/browse/HBASE-4412
Project: HBase
Issue Type: Bug
Reporter: Ming Ma
Assignee: Ming Ma
Fix For: 0.92.0
Attachments: HBASE-4412-trunk.patch
ScannerCallable.java doesn't retry on the same server in case of
NotServingRegionException.
if (ioe instanceof NotServingRegionException) {
// Throw a DNRE so that we break out of cycle of calling NSRE
// when what we need is to open scanner against new location.
// Attach NSRE to signal client that it needs to resetup scanner.
throw new DoNotRetryIOException(Reset scanner, ioe);
In the scenario when region server is in shutdown process,
RegionServerStoppedException will be thrown. ScannerCallable still retry. It
isn't necessary.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4145. --- Resolution: Fixed Hadoop Flags: Reviewed Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Fix For: 0.94.0 Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the MapReduce framework only performs sum on the counter values so it is tricky to find the max of certain metrics in all mapper instances. However, it might be good enough for now. With this approach, the metrics value will be available via MapReduce counter. a) Have ResultScanner return a new ResultScannerMetrics interface. b) TableInputFormat will access data from ResultScannerMetrics and populate MapReduce counters accordingly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
