[
https://issues.apache.org/jira/browse/HDFS-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122300#comment-14122300
]
Yang Jiandan commented on HDFS-6999:
------------------------------------
the stack is :
"regionserver60020-largeCompactions-1409055324582" daemon prio=10
tid=0x0000000001080800 nid=0x2c7c runnable [0x00000000601cb000]
java.lang.Thread.State: RUNNABLE
at org.apache.hadoop.net.unix.DomainSocket.readByteBufferDirect0(Native
Method)
at
org.apache.hadoop.net.unix.DomainSocket.access$400(DomainSocket.java:45)
at
org.apache.hadoop.net.unix.DomainSocket$DomainChannel.read(DomainSocket.java:628)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.readChannelFully(PacketReceiver.java:258)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:209)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:102)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:173)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:138)
- locked <0x000000047c41f7e0> (a
org.apache.hadoop.hdfs.RemoteBlockReader2)
at
org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682)
at
org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738)
- locked <0x00000004aaceca60> (a org.apache.hadoop.hdfs.DFSInputStream)
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836)
- locked <0x00000004aaceca60> (a org.apache.hadoop.hdfs.DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:563)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1215)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1430)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1312)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:392)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:643)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:757)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:136)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:507)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:217)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:76)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:109)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1086)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1480)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:475)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- <0x000000049e162b60> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x00000005974a84f0> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x000000065e45cf58> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
> PacketReceiver#readChannelFully is in an infinite loop
> ------------------------------------------------------
>
> Key: HDFS-6999
> URL: https://issues.apache.org/jira/browse/HDFS-6999
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, hdfs-client
> Affects Versions: 2.4.1
> Reporter: Yang Jiandan
> Priority: Critical
>
> In our cluster, we found hbase handler may be never return when it reads hdfs
> file using RemoteBlockReader2, and the hander thread occupys 100% cup. wo
> found this is because PacketReceiver#readChannelFully is in an infinite loop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
