[ https://issues.apache.org/jira/browse/HBASE-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727729#action_12727729 ]
stack commented on HBASE-1603: ------------------------------ The .meta. table is all wonky: {code} 2009-07-06 18:54:30,763 [RegionManager.metaScanner] INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {server: X.X.X.140:60020, regionname: .META.,,1, startKey: <>} 2009-07-06 18:54:30,768 [RegionManager.rootScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384', FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION => 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, serverName=X.X.X.140:60020, startCode=1246905530242 2009-07-06 18:54:30,769 [RegionManager.rootScanner] INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of 1 row(s) of meta region {server: X.X.X.142:60020, regionname: -ROOT-,,0, startKey: <>} complete 2009-07-06 18:54:30,771 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,,1246905732395', STARTKEY => '', ENDKEY => '\x00\x03\x02\x04\x07\x03\x07\x08\x00\x06', ENCODED => 340516641, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.142:60020, startCode=1246905530240 2009-07-06 18:54:30,773 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,,1246905998094', STARTKEY => '', ENDKEY => '\x00\x02\x03\x08\x09\x06\x05\x04\x03\x00', ENCODED => 815443872, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.142:60020, startCode=1246905530240 2009-07-06 18:54:30,775 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,,1246906356948', STARTKEY => '', ENDKEY => '\x00\x02\x03\x04\x09\x03\x03\x08\x01\x06', ENCODED => 660737141, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.142:60020, startCode=1246905530240 2009-07-06 18:54:30,776 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x02\x03\x04\x09\x03\x03\x08\x01\x06,1246906356948', STARTKEY => '\x00\x02\x03\x04\x09\x03\x03\x08\x01\x06', ENDKEY => '\x00\x02\x03\x08\x09\x06\x05\x04\x03\x00', ENCODED => 215524477, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.142:60020, startCode=1246905530240 2009-07-06 18:54:30,778 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x02\x03\x08\x09\x06\x05\x04\x03\x00,1246905998094', STARTKEY => '\x00\x02\x03\x08\x09\x06\x05\x04\x03\x00', ENDKEY => '\x00\x03\x02\x04\x07\x03\x07\x08\x00\x06', ENCODED => 1847085961, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.142:60020, startCode=1246905530240 2009-07-06 18:54:30,779 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x03\x02\x04\x06\x09\x01\x09\x05\x04,1246905800311', STARTKEY => '\x00\x03\x02\x04\x06\x09\x01\x09\x05\x04', ENDKEY => '\x00\x03\x07\x00\x03\x08\x04\x09\x09\x03', ENCODED => 2063095684, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.140:60020, startCode=1246905530242 2009-07-06 18:54:30,781 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x03\x02\x04\x07\x03\x07\x08\x00\x06,1246905800311', STARTKEY => '\x00\x03\x02\x04\x07\x03\x07\x08\x00\x06', ENDKEY => '\x00\x03\x02\x04\x06\x09\x01\x09\x05\x04', ENCODED => 398435244, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.140:60020, startCode=1246905530242 2009-07-06 18:54:30,782 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x03\x06\x05\x00\x09\x03\x03\x06\x07,1246905805297', STARTKEY => '\x00\x03\x06\x05\x00\x09\x03\x03\x06\x07', ENDKEY => '\x00\x03\x09\x02\x04\x09\x01\x01\x06\x07', ENCODED => 568573022, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.141:60020, startCode=1246905530271 2009-07-06 18:54:30,784 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x03\x07\x00\x03\x05\x09\x07\x03\x05,1246906209734', STARTKEY => '\x00\x03\x07\x00\x03\x05\x09\x07\x03\x05', ENDKEY => '\x00\x03\x06\x05\x00\x09\x03\x03\x06\x07', ENCODED => 1135953828, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.142:60020, startCode=1246905530240 2009-07-06 18:54:30,786 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x03\x07\x00\x03\x08\x04\x09\x09\x03,1246905647779', STARTKEY => '\x00\x03\x07\x00\x03\x08\x04\x09\x09\x03', ENDKEY => '\x00\x03\x09\x02\x04\x09\x01\x01\x06\x07', ENCODED => 1638479249, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.141:60020, startCode=1246905530271 2009-07-06 18:54:30,787 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x03\x07\x00\x03\x08\x04\x09\x09\x03,1246905805297', STARTKEY => '\x00\x03\x07\x00\x03\x08\x04\x09\x09\x03', ENDKEY => '\x00\x03\x06\x05\x00\x09\x03\x03\x06\x07', ENCODED => 585504103, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.140:60020, startCode=1246905530242 2009-07-06 18:54:30,788 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x03\x07\x00\x03\x08\x04\x09\x09\x03,1246906209734', STARTKEY => '\x00\x03\x07\x00\x03\x08\x04\x09\x09\x03', ENDKEY => '\x00\x03\x07\x00\x03\x05\x09\x07\x03\x05', ENCODED => 1683674128, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.140:60020, startCode=1246905530242 2009-07-06 18:54:30,790 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x03\x09\x02\x04\x09\x01\x01\x06\x07,1246905589105', STARTKEY => '\x00\x03\x09\x02\x04\x09\x01\x01\x06\x07', ENDKEY => '', ENCODED => 1426901695, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.141:60020, startCode=1246905530271 2009-07-06 18:54:30,791 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x03\x09\x02\x04\x09\x01\x01\x06\x07,1246905703044', STARTKEY => '\x00\x03\x09\x02\x04\x09\x01\x01\x06\x07', ENDKEY => '\x00\x04\x05\x02\x01\x09\x04\x09\x04\x07', ENCODED => 442661090, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.141:60020, startCode=1246905530271 2009-07-06 18:54:30,792 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x04\x05\x02\x01\x09\x04\x09\x04\x07,1246905703044', STARTKEY => '\x00\x04\x05\x02\x01\x09\x04\x09\x04\x07', ENDKEY => '', ENCODED => 1183734620, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.141:60020, startCode=1246905530271 2009-07-06 18:54:30,793 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x04\x05\x02\x01\x09\x04\x09\x04\x07,1246906206915', STARTKEY => '\x00\x04\x05\x02\x01\x09\x04\x09\x04\x07', ENDKEY => '\x00\x06\x05\x01\x05\x04\x07\x08\x05\x07', ENCODED => 1552614533, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.141:60020, startCode=1246905530271 2009-07-06 18:54:30,794 [RegionManager.metaScanner] DEBUG org.apache.hadoop.hbase.master.BaseScanner: ROW hri=REGION => {NAME => 'TestTable,\x00\x06\x05\x01\x05\x04\x07\x08\x05\x07,1246906206915', STARTKEY => '\x00\x06\x05\x01\x05\x04\x07\x08\x05\x07', ENDKEY => '', ENCODED => 373711036, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, serverName=X.X.X.141:60020, startCode=1246905530271 {code} End keys on one region do not match the start keys no next. > MR failed "RetriesExhaustedException: Trying to contact region server Some > server for region TestTable..." > ---------------------------------------------------------------------------------------------------------- > > Key: HBASE-1603 > URL: https://issues.apache.org/jira/browse/HBASE-1603 > Project: Hadoop HBase > Issue Type: Bug > Reporter: stack > Fix For: 0.20.0 > > > Here is the master. Region > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246462685358 was split at > 16:11:42,865. My MR job failed at 18:12:26,462 with this: > {code} > 2009-07-01 18:12:26,462 WARN org.apache.hadoop.mapred.TaskTracker: Error > running child > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact > region server Some server for region TestTable,�,1246464670313, row '�� > ', but failed after 10 attempts. > Exceptions: > ... > {code} > Why after ten attempts did the client not find the region? > {code} > 2009-07-01 16:11:42,865 [IPC Server handler 2 on 60001] INFO > org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246462685358: Daughters; > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313, > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from > aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3 > 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO > org.apache.hadoop.hbase.master.RegionManager: Assigning region > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 to > aa0-000-15.u.powerset.com,60020,1246461673026 > 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO > org.apache.hadoop.hbase.master.RegionManager: Assigning region > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 to > aa0-000-15.u.powerset.com,60020,1246461673026 > 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO > org.apache.hadoop.hbase.master.ServerManager: Received > MSG_REPORT_PROCESS_OPEN: > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from > aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3 > 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO > org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 from > aa0-000-15.u.powerset.com,60020,1246461673026; 2 of 3 > 2009-07-01 16:11:45,906 [IPC Server handler 8 on 60001] INFO > org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from > aa0-000-15.u.powerset.com,60020,1246461673026; 3 of 3 > 2009-07-01 16:11:45,906 [HMaster] INFO > org.apache.hadoop.hbase.master.RegionServerOperation: > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 open on > 208.76.44.142:60020 > 2009-07-01 16:11:45,906 [HMaster] INFO > org.apache.hadoop.hbase.master.RegionServerOperation: updating row > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 in region > .META.,,1 with startcode 1246461673026 and server 208.76.44.142:60020 > 2009-07-01 16:11:45,908 [HMaster] INFO > org.apache.hadoop.hbase.master.RegionServerOperation: > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 open on > 208.76.44.142:60020 > 2009-07-01 16:11:45,908 [HMaster] INFO > org.apache.hadoop.hbase.master.RegionServerOperation: updating row > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 in region > .META.,,1 with startcode 1246461673026 and server 208.76.44.142:60020 > 2009-07-01 17:46:42,670 [IPC Server handler 0 on 60001] INFO > org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313: Daughters; > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246470379467, > TestTable,\x00\x00\x08\x04\x05\x07\x02\x05\x04\x08,1246470379467 from > aa0-000-15.u.powerset.com,60020,1246461673026; 5 of 7 > {code} > Here is over on the regionserver: > {code} > 2009-07-01 16:11:42,865 [IPC Server handler 2 on 60001] INFO > org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246462685358: Daughters; > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313, > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from > aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3 > 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO > org.apache.hadoop.hbase.master.RegionManager: Assigning region > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 to > aa0-000-15.u.powerset.com,60020,1246461673026 > 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO > org.apache.hadoop.hbase.master.RegionManager: Assigning region > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 to > aa0-000-15.u.powerset.com,60020,1246461673026 > 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO > org.apache.hadoop.hbase.master.ServerManager: Received > MSG_REPORT_PROCESS_OPEN: > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from > aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3 > 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO > org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 from > aa0-000-15.u.powerset.com,60020,1246461673026; 2 of 3 > 2009-07-01 16:11:45,906 [IPC Server handler 8 on 60001] INFO > org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN: > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from > aa0-000-15.u.powerset.com,60020,1246461673026; 3 of 3 > 2009-07-01 16:11:45,906 [HMaster] INFO > org.apache.hadoop.hbase.master.RegionServerOperation: > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 open on > X.X.X.142:60020 > 2009-07-01 16:11:45,906 [HMaster] INFO > org.apache.hadoop.hbase.master.RegionServerOperation: updating row > TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 in region > .META.,,1 with startcode 1246461673026 and server X.X.X4.142:60020 > 2009-07-01 16:11:45,908 [HMaster] INFO > org.apache.hadoop.hbase.master.RegionServerOperation: > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 open on > X.X.X.142:60020 > 2009-07-01 16:11:45,908 [HMaster] INFO > org.apache.hadoop.hbase.master.RegionServerOperation: updating row > TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 in region > .META.,,1 with startcode 1246461673026 and server X.X.X.142:60020 > {code} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.