Stuck on memcache flush
-----------------------
Key: HBASE-1415
URL: https://issues.apache.org/jira/browse/HBASE-1415
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Fix For: 0.20.0
Latest trunk, regionserver gets stuck doing below:
{code}
2009-05-13 03:05:28,126 [regionserver/0:0:0:0:0:0:0:0:60021.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Closed
hdfs://aa0-000-12.u.powerset.com:9000/hbasetrunk2/.logs/aa0-000-13.u.powerset.com_1242178588540_60021/hlog.dat.1242183899413,
entries=10051
2009-05-13 03:05:28,126 [regionserver/0:0:0:0:0:0:0:0:60021.logRoller] DEBUG
org.apache.hadoop.hbase.regionserver.HLog: Found 0 logs to remove out of total
53; oldest outstanding seqnum is 6838308 from region
TestTable,0842399686,1242180442452
2009-05-13 03:05:28,737 [IPC Server handler 9 on 60021] INFO
org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Forced flushing of
TestTable,0107641622,1242181965182 because global memcache limit of 497.8m
exceeded; currently 513.3m and flushing till 311.1m
2009-05-13 03:05:28,737 [IPC Server handler 9 on 60021] DEBUG
org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Too many store files in
store info: 5, waiting
2009-05-13 03:05:28,737 [IPC Server handler 9 on 60021] DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
for region TestTable,0107641622,1242181965182/614435386 because:
regionserver/0:0:0:0:0:0:0:0:60021.cacheFlusher
2009-05-13 03:06:07,143 [regionserver/0:0:0:0:0:0:0:0:60021.compactor] DEBUG
org.apache.hadoop.hbase.regionserver.Store: Completed compaction of info; store
size is 168.0m
2009-05-13 03:06:07,143 [regionserver/0:0:0:0:0:0:0:0:60021.compactor] INFO
org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
TestTable,0092188538,1242183375224 in 3mins, 58sec
2009-05-13 03:06:58,757 [IPC Server handler 9 on 60021] DEBUG
org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Too many store files in
store info: 5, waiting
2009-05-13 03:06:58,757 [IPC Server handler 9 on 60021] DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
for region TestTable,0107641622,1242181965182/614435386 because:
regionserver/0:0:0:0:0:0:0:0:60021.cacheFlusher
2009-05-13 03:08:28,767 [IPC Server handler 9 on 60021] DEBUG
org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Too many store files in
store info: 5, waiting
2009-05-13 03:08:28,767 [IPC Server handler 9 on 60021] DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
for region TestTable,0107641622,1242181965182/614435386 because:
regionserver/0:0:0:0:0:0:0:0:60021.cacheFlusher
2009-05-13 03:09:58,777 [IPC Server handler 9 on 60021] DEBUG
org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Too many store files in
store info: 5, waiting
...
{code}
Thread dumping I see a few threads blocked here:
{code}
"IPC Server handler 4 on 60021" daemon prio=10 tid=0x00007f42982ee000
nid=0x599b waiting for monitor entry [0x000000004380b000..0x000000004380bb00]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:294)
- waiting to lock <0x00007f42aac8cbd8> (a
org.apache.hadoop.hbase.regionserver.MemcacheFlusher)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1800)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:642)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:911)
..
{code}
And this is a good while after my MR upload failed.
Assigning myself since I can reproduce easy enough.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
