[
https://issues.apache.org/jira/browse/HBASE-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729795#action_12729795
]
stack commented on HBASE-1603:
------------------------------
I committed this debugging patch. Will help if we see this issue again. Also
had fix up for when exceptions throw with messages that have binary data in it
and some small javadoc fixes.
> MR failed "RetriesExhaustedException: Trying to contact region server Some
> server for region TestTable..."
> ----------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1603
> URL: https://issues.apache.org/jira/browse/HBASE-1603
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: debugging-v4.patch
>
>
> Here is the master. Region
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246462685358 was split at
> 16:11:42,865. My MR job failed at 18:12:26,462 with this:
> {code}
> 2009-07-01 18:12:26,462 WARN org.apache.hadoop.mapred.TaskTracker: Error
> running child
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
> region server Some server for region TestTable,�,1246464670313, row '��
> ', but failed after 10 attempts.
> Exceptions:
> ...
> {code}
> Why after ten attempts did the client not find the region?
> {code}
> 2009-07-01 16:11:42,865 [IPC Server handler 2 on 60001] INFO
> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT:
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246462685358: Daughters;
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313,
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from
> aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3
> 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO
> org.apache.hadoop.hbase.master.RegionManager: Assigning region
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 to
> aa0-000-15.u.powerset.com,60020,1246461673026
> 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO
> org.apache.hadoop.hbase.master.RegionManager: Assigning region
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 to
> aa0-000-15.u.powerset.com,60020,1246461673026
> 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO
> org.apache.hadoop.hbase.master.ServerManager: Received
> MSG_REPORT_PROCESS_OPEN:
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from
> aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3
> 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO
> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 from
> aa0-000-15.u.powerset.com,60020,1246461673026; 2 of 3
> 2009-07-01 16:11:45,906 [IPC Server handler 8 on 60001] INFO
> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from
> aa0-000-15.u.powerset.com,60020,1246461673026; 3 of 3
> 2009-07-01 16:11:45,906 [HMaster] INFO
> org.apache.hadoop.hbase.master.RegionServerOperation:
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 open on
> 208.76.44.142:60020
> 2009-07-01 16:11:45,906 [HMaster] INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 in region
> .META.,,1 with startcode 1246461673026 and server 208.76.44.142:60020
> 2009-07-01 16:11:45,908 [HMaster] INFO
> org.apache.hadoop.hbase.master.RegionServerOperation:
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 open on
> 208.76.44.142:60020
> 2009-07-01 16:11:45,908 [HMaster] INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 in region
> .META.,,1 with startcode 1246461673026 and server 208.76.44.142:60020
> 2009-07-01 17:46:42,670 [IPC Server handler 0 on 60001] INFO
> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT:
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313: Daughters;
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246470379467,
> TestTable,\x00\x00\x08\x04\x05\x07\x02\x05\x04\x08,1246470379467 from
> aa0-000-15.u.powerset.com,60020,1246461673026; 5 of 7
> {code}
> Here is over on the regionserver:
> {code}
> 2009-07-01 16:11:42,865 [IPC Server handler 2 on 60001] INFO
> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT:
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246462685358: Daughters;
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313,
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from
> aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3
> 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO
> org.apache.hadoop.hbase.master.RegionManager: Assigning region
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 to
> aa0-000-15.u.powerset.com,60020,1246461673026
> 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO
> org.apache.hadoop.hbase.master.RegionManager: Assigning region
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 to
> aa0-000-15.u.powerset.com,60020,1246461673026
> 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO
> org.apache.hadoop.hbase.master.ServerManager: Received
> MSG_REPORT_PROCESS_OPEN:
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from
> aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3
> 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO
> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 from
> aa0-000-15.u.powerset.com,60020,1246461673026; 2 of 3
> 2009-07-01 16:11:45,906 [IPC Server handler 8 on 60001] INFO
> org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 from
> aa0-000-15.u.powerset.com,60020,1246461673026; 3 of 3
> 2009-07-01 16:11:45,906 [HMaster] INFO
> org.apache.hadoop.hbase.master.RegionServerOperation:
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 open on
> X.X.X.142:60020
> 2009-07-01 16:11:45,906 [HMaster] INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 in region
> .META.,,1 with startcode 1246461673026 and server X.X.X4.142:60020
> 2009-07-01 16:11:45,908 [HMaster] INFO
> org.apache.hadoop.hbase.master.RegionServerOperation:
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 open on
> X.X.X.142:60020
> 2009-07-01 16:11:45,908 [HMaster] INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: updating row
> TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 in region
> .META.,,1 with startcode 1246461673026 and server X.X.X.142:60020
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
