[
https://issues.apache.org/jira/browse/HBASE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-634.
-------------------------
Resolution: Won't Fix
In testing, PE runs about 15% slower with this patch installed. Comparing
length of compaction runs over life of the job, seems like compacting to local
can take a little longer.
Using local:
{code}
2008-05-22 00:27:21,182 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region -ROOT-,,0 in 0sec
2008-05-22 00:27:27,070 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region .META.,,1 in 0sec
2008-05-22 00:28:33,175 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211416112908 in 0sec
2008-05-22 00:29:09,960 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211416112908 in 0sec
2008-05-22 00:29:34,487 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211416112908 in 0sec
2008-05-22 00:30:14,497 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211416112908 in 16sec
2008-05-22 00:30:30,216 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211416112908 in 0sec
2008-05-22 00:30:46,616 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000097920,1211416230248 in 13sec
2008-05-22 00:30:57,270 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211416230247 in 10sec
2008-05-22 00:31:11,094 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000097920,1211416230248 in 0sec
2008-05-22 00:31:59,049 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000097920,1211416230248 in 26sec
2008-05-22 00:32:16,873 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000244992,1211416319084 in 13sec
2008-05-22 00:32:29,826 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000097920,1211416319083 in 12sec
2008-05-22 00:32:46,885 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000244992,1211416319084 in 0sec
2008-05-22 00:33:35,292 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000244992,1211416319084 in 25sec
2008-05-22 00:33:51,602 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000244992,1211416415301 in 11sec
2008-05-22 00:34:09,929 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000401920,1211416415302 in 18sec
2008-05-22 00:34:21,506 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000401920,1211416415302 in 0sec
2008-05-22 00:35:07,642 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000401920,1211416415302 in 23sec
2008-05-22 00:35:26,509 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000559232,1211416507646 in 16sec
2008-05-22 00:35:39,750 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000401920,1211416507645 in 13sec
2008-05-22 00:35:54,023 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000559232,1211416507646 in 0sec
2008-05-22 00:36:44,078 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000559232,1211416507646 in 26sec
2008-05-22 00:36:56,356 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000559232,1211416604090 in 10sec
2008-05-22 00:37:14,006 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000718720,1211416604090 in 17sec
2008-05-22 00:37:29,184 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000718720,1211416604090 in 0sec
2008-05-22 00:38:16,196 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000718720,1211416604090 in 25sec
2008-05-22 00:38:32,634 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000877696,1211416696222 in 13sec
2008-05-22 00:38:45,692 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,000718720,1211416696222 in 13sec
{code}
Going against HDFS:
{code}
2008-05-22 00:14:40,749 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region -ROOT-,,0
2008-05-22 00:14:40,751 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region -ROOT-,,0 in 0sec
2008-05-22 00:14:46,639 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region .META.,,1
2008-05-22 00:14:46,643 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region .META.,,1 in 0sec
2008-05-22 00:15:04,664 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,,1211415303994
2008-05-22 00:15:04,667 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211415303994 in 0sec
2008-05-22 00:15:41,517 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,,1211415303994
2008-05-22 00:15:41,520 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211415303994 in 0sec
2008-05-22 00:16:06,026 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,,1211415303994
2008-05-22 00:16:06,028 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211415303994 in 0sec
2008-05-22 00:16:29,056 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,,1211415303994
2008-05-22 00:16:38,339 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211415303994 in 9sec
2008-05-22 00:16:56,149 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,,1211415303994
2008-05-22 00:16:56,162 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211415303994 in 0sec
2008-05-22 00:16:58,969 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,,1211415416165
2008-05-22 00:17:04,818 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,,1211415416165 in 5sec
2008-05-22 00:17:04,818 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415416166
2008-05-22 00:17:12,902 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415416166 in 8sec
2008-05-22 00:17:32,972 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415416166
2008-05-22 00:17:32,973 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415416166 in 0sec
2008-05-22 00:17:53,673 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415416166
2008-05-22 00:18:07,002 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415416166 in
13sec
2008-05-22 00:18:11,145 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415487020
2008-05-22 00:18:16,352 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415487020 in 5sec
2008-05-22 00:18:16,352 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415487021
2008-05-22 00:18:23,901 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415487021 in 7sec
2008-05-22 00:18:45,131 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415487021
2008-05-22 00:18:45,132 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415487021 in 0sec
2008-05-22 00:19:09,027 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415487021
2008-05-22 00:19:22,156 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415487021 in
13sec
2008-05-22 00:19:26,277 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415562160
2008-05-22 00:19:33,813 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415562160 in 7sec
2008-05-22 00:19:33,813 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415562161
2008-05-22 00:19:43,012 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415562161 in 9sec
2008-05-22 00:20:00,202 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415562161
2008-05-22 00:20:00,203 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415562161 in 0sec
2008-05-22 00:20:24,163 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415562161
2008-05-22 00:20:37,632 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415562161 in
13sec
2008-05-22 00:20:41,505 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415637637
2008-05-22 00:20:50,317 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415637637 in 8sec
2008-05-22 00:20:50,318 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415637637
2008-05-22 00:20:59,944 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415637637 in 9sec
2008-05-22 00:21:17,215 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415637637
2008-05-22 00:21:17,216 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415637637 in 0sec
2008-05-22 00:21:40,891 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415637637
2008-05-22 00:21:57,139 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415637637 in
16sec
2008-05-22 00:21:59,652 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415717149
2008-05-22 00:22:08,234 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415717149 in 8sec
2008-05-22 00:22:08,235 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415717150
2008-05-22 00:22:17,192 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415717150 in 8sec
2008-05-22 00:22:37,439 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415717150
2008-05-22 00:22:37,440 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415717150 in 0sec
2008-05-22 00:23:01,107 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415717150
2008-05-22 00:23:17,565 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415717150 in
16sec
2008-05-22 00:23:20,782 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415797573
2008-05-22 00:23:29,291 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415797573 in 8sec
2008-05-22 00:23:29,291 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region TestTable,[EMAIL PROTECTED],1211415797574
2008-05-22 00:23:39,535 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,[EMAIL PROTECTED],1211415797574 in
10sec
{code}
Not a very scienttific comparison but would think that compacting local would
have to be significantly faster if we were to add this patch. Closing as 'wont
fix'.
> hbase hungup on hdfs: appending to HLog. Won't shut down.
> ----------------------------------------------------------
>
> Key: HBASE-634
> URL: https://issues.apache.org/jira/browse/HBASE-634
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.2.0, 0.1.2
>
>
> On our internal cluster, noticed hung regionserver. Hang manifest itself in
> log as thousands of lines of:
> {code}
> Call queue overflow discarding oldest call batchUpdate
> {code}
> Thread dumping, a bunch of threads are waiting to append to HLog:
> {code}
> 41 "IPC Server handler 8 on 60020" daemon prio=1 tid=0x00002aab40226770
> nid=0x3890 waiting for monitor entry [0x0000000042d7d000..0x0000000042d7db00]
> 42 at org.apache.hadoop.hbase.HLog.append(HLog.java:370)
> 43 - waiting to lock <0x00002aaab7815d38> (a java.lang.Integer)
> 44 at org.apache.hadoop.hbase.HRegion.update(HRegion.java:1624)
> 45 at
> org.apache.hadoop.hbase.HRegion.batchUpdate(HRegion.java:1427)
> 46 at
> org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1554)
> 47 at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown
> Source)
> 48 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> 49 at java.lang.reflect.Method.invoke(Unknown Source)
> 50 at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> 51 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
> {code}
> ... but they can't because another thread is stuck trying to write the HLog:
> {code}
> 16 "IPC Server handler 9 on 60020" daemon prio=1 tid=0x00002aab402278d0
> nid=0x3891 in Object.wait() [0x0000000042e7e000..0x0000000042e7eb80]
> 17 at java.lang.Object.wait(Native Method)
> 18 at java.lang.Object.wait(Unknown Source)
> 19 at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:2131)
> 20 - locked <0x00002aaab7ee5038> (a java.util.LinkedList)
> 21 at
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:141)
> 22 at
> org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
> 23 at
> org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> 24 - locked <0x00002aaab7ee4cb0> (a
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream)
> 25 at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:41)
> 26 at java.io.DataOutputStream.write(Unknown Source)
> 27 - locked <0x00002aaab7e73ea8> (a
> org.apache.hadoop.fs.FSDataOutputStream)
> 28 at
> org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:990)
> 29 - locked <0x00002aaab7ee5200> (a
> org.apache.hadoop.io.SequenceFile$Writer)
> 30 at org.apache.hadoop.hbase.HLog.append(HLog.java:387)
> 31 - locked <0x00002aaab7815d38> (a java.lang.Integer)
> 32 at org.apache.hadoop.hbase.HRegion.update(HRegion.java:1624)
> 33 at
> org.apache.hadoop.hbase.HRegion.batchUpdate(HRegion.java:1427)
> 34 at
> org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1554)
> 35 at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown
> Source)
> 36 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> 37 at java.lang.reflect.Method.invoke(Unknown Source)
> 38 at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> 39 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
> {code}
> Looking in code, the above sleep will be woken when we get response from
> datanode -- a response that never comes in this case. The Responder thread
> itself is stuck trying to read a long from the datanode:
> {code}
> 3 "ResponseProcessor for block blk_3392187502501092232" daemon prio=1
> tid=0x00002aab38cd8ba0 nid=0x7700 runnable
> [0x0000000043080000..0x0000000043080c80]
> 4 at java.net.SocketInputStream.socketRead0(Native Method)
> 5 at java.net.SocketInputStream.read(Unknown Source)
> 6 at java.io.DataInputStream.readFully(Unknown Source)
> 7 at java.io.DataInputStream.readLong(Unknown Source)
> 8 at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:1734)
> {code}
> Related is the DFSClient DataStreamer, itself is in a sleep
> {code}
> 10 "DataStreamer for file
> /hbase/aa0-005-2.u.powerset.com/log_208.76.44.96_1211224091595_60020/hlog.dat.004"
> daemon prio=1 tid=0x00002aab38a34920 nid=0x6e1b in Object.wait()
> [0x0000000043484000..0x0000000043484b00]
> 11 at java.lang.Object.wait(Native Method)
> 12 at java.lang.Object.wait(Unknown Source)
> 13 at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1656)
> 14 - locked <0x00002aaab7ee5060> (a java.util.LinkedList)
> {code}
> The hang doesn't change after 5 or 6 thread dumps nor does it change though I
> shutdown the regionserver.
> Would need to figure why the datanode stopped responding, why we haven't
> timedout our read at least.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
