As part of HADOOP-2346, I changed how default socketFactory creates its
socket : from 'new Socket()' to 'SocketChannel.open().socket()' since
DFSClient uses socket factory. As a side effect, ipc.Client ends up
using this socket creation.
ipc.Client's internal response receiver is blocked on
socket.getInputStream().read() as expected. But other threads can not
send RPCs since their write() is blocked on a socket imlementation
related lock. This lock is held by the reader. Thats pretty weird. How
can it not allow read and write at the same time?
Raghu.
Related stack traces (from a Junit test):
read : locked 0xd32b6bf0
=============================
"IPC Client connection to localhost/127.0.0.1:55567" daemon prio=10
tid=0xc02dfc00 nid=0x2c41 runnable [0xbf836000..0xbf836fb0]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:184)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xd32c30c8> (a sun.nio.ch.Util$1)
- locked <0xd32c30b8> (a java.util.Collections$UnmodifiableSet)
- locked <0xd32c2ed0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:193)
- locked <0xd32b6bf0> (a java.lang.Object)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
- locked <0xd32b6f50> (a sun.nio.ch.SocketAdaptor$SocketInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.hadoop.ipc.Client$Connection$1.read(Client.java:190)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked <0xd32b6f10> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:276)
write : trying to lock 0xd32b6bf0
==================================
"main" prio=10 tid=0x0805b000 nid=0x2b85 waiting for monitor entry
[0xd7e66000..0xd7e67288]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.nio.channels.Channels.write(Channels.java:58)
- waiting to lock <0xd32b6bf0> (a java.lang.Object)
at java.nio.channels.Channels.access$000(Channels.java:47)
at java.nio.channels.Channels$1.write(Channels.java:134)
- locked <0xd32b90f8> (a java.nio.channels.Channels$1)
at org.apache.hadoop.ipc.Client$Connection$2.write(Client.java:201)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0xd32b90c0> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:346)
- locked <0xd32b90a8> (a java.io.DataOutputStream)
at org.apache.hadoop.ipc.Client.call(Client.java:504)
- locked <0xd32bb3b0> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
at org.apache.hadoop.dfs.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:291)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:278)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:315)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:260)
at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:211)
at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:166)
at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:2532)
at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2476)
at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2497)
at
org.apache.hadoop.dfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:298)
