[
https://issues.apache.org/jira/browse/HBASE-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010526#comment-13010526
]
stack commented on HBASE-3154:
------------------------------
This is what it looks like on server-side:
{code}
2011-03-23 16:37:08,498 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call multi(org.apache.hadoop.hbase.client.MultiAction@6db16bc) from
10.4.8.34:42287: output error
2011-03-23 16:37:08,498 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 61020 caught: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at org.apache.hadoop.hbase.ipc.HBaseServer.channelIO(HBaseServer.java:1387)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1339)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
2011-03-23 16:37:08,515 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call multi(org.apache.hadoop.hbase.client.MultiAction@5754a4a7) from
10.4.8.34:42287: output error
2011-03-23 16:37:08,516 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 61020 caught: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at org.apache.hadoop.hbase.ipc.HBaseServer.channelIO(HBaseServer.java:1387)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1339)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
{code}
On client side it looks like this:
{code}
11/03/23 17:03:16 DEBUG client.HConnectionManager$HConnectionImplementation:
Failed all from sv4borg230:61020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Call
to sv4borg230/10.4.12.34:61020 failed on socket timeout exception:
java.net.SocketTimeoutException: 1000 millis timeout while waiting for channel
to
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1158)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1246)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:822)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:678)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:663)
at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:316)
at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:390)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:155)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:372)
at com.yahoo.ycsb.ClientThread.run(Client.java:251)
Caused by: java.net.SocketTimeoutException: Call to sv4borg230/10.4.12.34:61020
failed on socket timeout exception: java.net.SocketTimeoutException: 1000
millis timeout while waiting for channel to be ready for read. ch : java.
at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy5.multi(Unknown Source)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4$1.call(HConnectionManager.java:1078)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4$1.call(HConnectionManager.java:1076)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1035)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1075)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1073)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: 1000 millis timeout while waiting
for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.4.8.34:37568
remote=sv4borg230/10.4.12.34:61020]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:299)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:539)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:477)
{code}
> HBase RPC should support timeout
> --------------------------------
>
> Key: HBASE-3154
> URL: https://issues.apache.org/jira/browse/HBASE-3154
> Project: HBase
> Issue Type: Improvement
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.90.0
>
> Attachments: hbaseRpcTimeoutTrunk.patch
>
>
> We see a client stuck at an RPC to a zombie RS. HBase RPC should support
> timeout.
> "IPC Client (47) connection to
> msgstore179.ash2.facebook.com/10.138.89.182:60020 from an unknown user"
> daemon prio=10 tid=0x00007f554c5ab800 nid=0x5df3 runnable [0x00000000590d4000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00007f5aa402a9e8> (a sun.nio.ch.Util$1)
> - locked <0x00007f5aa402aac0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00007f596a398408> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.FilterInputStream.read(FilterInputStream.java:116)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:279)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> - locked <0x00007f5a1d7e7bd8> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.readInt(DataInputStream.java:370)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:514)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:454)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
