Hi, There,
I have been working with hbase 0.20.4 and it is very unstable, constantly
dropping regions in the master
causing problem .
This morning, I upgraded to 0.20.5 , however, I got some new problem, here is
the result on the client side:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact regi
on server Some server, retryOnlyOne=true, index=0, islastrow=false, tries=9, num
tries=10, i=0, listsize=11, region=Spam_MsgEventTable,2010-06-28 11:34:02\x09064
dcbadda64986a6caf49b7eb31572a,1277730384014 for region Spam_MsgEventTable,2010-0
6-28 11:34:02\x09064dcbadda64986a6caf49b7eb31572a,1277730384014, row '2010-06-28
20:13:08\x0915eb35c0f8bffe58939cf59fa1106363', but failed after 10 attempts.
Exceptions:
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.
process(HConnectionManager.java:1167)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.proces
sBatchOfRows(HConnectionManager.java:1248)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666)
at com.proofpoint.prscollect.loader.MsgEventCollector$MsgEventCollectorM
apper.map(MsgEventCollector.java:406)
at com.proofpoint.prscollect.loader.MsgEventCollector$MsgEventCollectorM
apper.map(MsgEventCollector.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:159)
Here is the result on the master:
2010-06-28 17:47:08,286 INFO org.apache.hadoop.hbase.master.ServerManager: Proce
ssing MSG_REPORT_PROCESS_OPEN: Spam_MsgEventTable,2010-06-28 11:34:02\x09064dcba
dda64986a6caf49b7eb31572a,1277730384014 from m0002035.ppops.net,60020,1277747217
286; 3 of 13
2010-06-28 17:47:08,434 INFO org.apache.hadoop.hbase.master.ServerManager: Proce
ssing MSG_REPORT_PROCESS_OPEN: Spam_MsgEventTable,2010-06-28 11:34:02\x09064dcba
dda64986a6caf49b7eb31572a,1277730384014 from m0002035.ppops.net,60020,1277747217
286; 2 of 12
2010-06-28 17:47:10,156 INFO org.apache.hadoop.hbase.master.ServerManager: Proce
ssing MSG_REPORT_OPEN: Spam_MsgEventTable,2010-06-28 11:34:02\x09064dcbadda64986
a6caf49b7eb31572a,1277730384014 from m0002035.ppops.net,60020,1277747217286; 1 o
f 10
2010-06-28 17:47:10,156 INFO org.apache.hadoop.hbase.master.RegionServerOperatio
n: Spam_MsgEventTable,2010-06-28 11:34:02\x09064dcbadda64986a6caf49b7eb31572a,12
77730384014 open on 10.110.8.85:60020
2010-06-28 17:47:10,157 INFO org.apache.hadoop.hbase.master.RegionServerOperatio
n: Updated row Spam_MsgEventTable,2010-06-28 11:34:02\x09064dcbadda64986a6caf49b
7eb31572a,1277730384014 in region .META.,,1 with startcode=1277747217286, server
=10.110.8.85:60020
Here is the result on the region server for the same region:
hbase-hadoop-regionserver-m0002035.log:2010-06-28 20:36:53,243 DEBUG org.apache.
hadoop.hbase.regionserver.HRegionServer: Batch puts interrupted at index=0 becau
se:Requested row out of range for HRegion Spam_MsgEventTable,2010-06-28 11:34:02
\x09064dcbadda64986a6caf49b7eb31572a,1277730384014, startKey='2010-06-28 11:34:0
2\x09064dcbadda64986a6caf49b7eb31572a', getEndKey()='2010-06-28 12:30:53\x09106e
01c383d237d7e44b4af237cd8450', row='2010-06-28 20:10:04\x0900c4fc3db8be8592b89fe
a0ee9a59265'
hbase-hadoop-regionserver-m0002035.log:2010-06-28 20:36:55,118 DEBUG org.apache.
hadoop.hbase.regionserver.HRegionServer: Batch puts interrupted at index=0 becau
se:Requested row out of range for HRegion Spam_MsgEventTable,2010-06-28 11:34:02
\x09064dcbadda64986a6caf49b7eb31572a,1277730384014, startKey='2010-06-28 11:34:0
2\x09064dcbadda64986a6caf49b7eb31572a', getEndKey()='2010-06-28 12:30:53\x09106e
01c383d237d7e44b4af237cd8450', row='2010-06-28 20:09:52\x090026210308b7e48c4764b
1939fdd8e56'
hbase-hadoop-regionserver-m0002035.log:2010-06-28 20:36:55,878 DEBUG org.apache.
hadoop.hbase.regionserver.HRegionServer: Batch puts interrupted at index=0 becau
se:Requested row out of range for HRegion Spam_MsgEventTable,2010-06-28 11:34:02
\x09064dcbadda64986a6caf49b7eb31572a,1277730384014, startKey='2010-06-28 11:34:0
2\x09064dcbadda64986a6caf49b7eb31572a', getEndKey()='2010-06-28 12:30:53\x09106e
01c383d237d7e44b4af237cd8450', row='2010-06-28 20:06:02\x09000273557fb30479ffb0a
bbee5446239'
Does anybody know what to do to get this hbase usable ?
Jimmy.
