[ https://issues.apache.org/jira/browse/HBASE-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-1992. -------------------------- Resolution: Invalid No longer valid. Splits are noted in zk now. On split, when new master comes online, it does fixup if the daughters were not properly noted in .META. as part of split (A unit test proves this latter facility) > We give up reporting splits if .META. is offline or unreachable (handlers at > their maximum) > ------------------------------------------------------------------------------------------- > > Key: HBASE-1992 > URL: https://issues.apache.org/jira/browse/HBASE-1992 > Project: HBase > Issue Type: Bug > Reporter: stack > > Here is our fail reporting a split. Lets make sure that this kinda event is > not dropped as part of master rewrite > {code} > 2009-11-17 20:57:51,943 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767/459329844 available; > sequence id is 25187748 > 2009-11-17 20:57:51,944 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767 > 2009-11-17 20:57:52,007 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META > updated, and report to master all successful. Old region=REGION => {NAME => > 'dds-test,j\xFD\xD7e\x8B\x1 > 0\xFC\xFE,1258446041463', STARTKEY => 'j\xFD\xD7e\x8B\x10\xFC\xFE', ENDKEY => > 'k\x27q6\xCA\xC8\x15,', ENCODED => 1069248233, OFFLINE => true, SPLIT => > true, TABLE => {{NAME => 'dds-test', FAMILIES => [{NAME = > > 'data', COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', > > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => > > 'meta', COMPRESSION => 'NONE', VERSIONS => '1', TTL => '21474 > 83647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, > new regions: dds-test,j\xFD\xD7e\x8B\x10\xFC\xFE,1258520203767, > dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767. Split took 1min > s, 8sec > 2009-11-17 20:57:52,007 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region > dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505 > 2009-11-17 20:57:54,779 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767 > 2009-11-17 20:57:54,780 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: > dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767 > 2009-11-17 20:57:54,864 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767/459329844 available; > sequence id is 25187748 > 2009-11-17 20:58:16,166 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC > Server handler 6 on 60020 took 2967ms appending an edit to hlog; editcount=58 > 2009-11-17 20:58:16,166 WARN org.apache.hadoop.hbase.regionserver.HLog: > regionserver/10.10.0.180:60020.logFlusher took 2854ms optional sync'ing hlog; > editcount=59 > 2009-11-17 20:58:34,340 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC > Server handler 6 on 60020 took 7615ms appending an edit to hlog; editcount=820 > 2009-11-17 20:58:34,340 WARN org.apache.hadoop.hbase.regionserver.HLog: > regionserver/10.10.0.180:60020.logFlusher took 1019ms optional sync'ing hlog; > editcount=821 > 2009-11-17 21:00:19,366 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505 in 2mins, 27sec > 2009-11-17 21:00:19,367 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting split of region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505 > 2009-11-17 21:00:22,237 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505 > 2009-11-17 21:00:32,829 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258520419408/538348405 > available; sequence id is 25188602 > 2009-11-17 21:00:32,829 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258520419408 > 2009-11-17 21:00:33,002 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region dds-test,\x7C\xAD\xCF\x27\x93\x10\x9A5,1258520419408/1604966836 > available; sequence id is 25188603 > 2009-11-17 21:00:33,003 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed dds-test,\x7C\xAD\xCF\x27\x93\x10\x9A5,1258520419408 > 2009-11-17 21:01:51,211 ERROR > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction/Split > failed for region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505 > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact > region server 10.10.0.189:60020 for region .META.,,1, row > 'dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505', but failed after 1 > 0 attempts. > Exceptions: > java.net.ConnectException: Connection refused > java.net.ConnectException: Connection refused > java.net.ConnectException: Connection refused > java.net.ConnectException: Connection refused > java.net.ConnectException: Connection refused > java.net.ConnectException: Connection refused > java.net.ConnectException: Connection refused > java.net.ConnectException: Connection refused > java.net.ConnectException: Connection refused > java.net.ConnectException: Connection refused > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1001) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.doCall(HConnectionManager.java:1192) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1114) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1200) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:605) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:470) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:211) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:106) > 2009-11-17 21:02:00,709 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region dds-test,d\xC7s\x1B\xD3\x23we,1258519740878 > 2009-11-17 21:02:11,147 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region dds-test,d\xC7s\x1B\xD3\x23we,1258519740878 in > 10sec > 2009-11-17 21:02:11,147 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region dds-test,a\x96\x85\xB6fV\xE1\xFB,1258519760278 > 2009-11-17 21:02:30,950 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll > /hbase/.logs/sl0127,60020,1258512074419/hlog.dat.1258520252642, entries=853, > calcsize=1428860, filesize=1342144. New hlog /hbase/.l > ogs/sl0127,60020,1258512074419/hlog.dat.1258520550914 > 2009-11-17 21:02:30,950 WARN org.apache.hadoop.hbase.regionserver.HLog: > regionserver/10.10.0.180:60020.logFlusher took 7413ms optional sync'ing hlog; > editcount=0 > 2009-11-17 21:03:29,990 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region dds-test,a\x96\x85\xB6fV\xE1\xFB,1258519760278 > in 1mins, 18sec > {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira