[
https://issues.apache.org/jira/browse/HBASE-28156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776229#comment-17776229
]
Duo Zhang commented on HBASE-28156:
-----------------------------------
{noformat}
"RS-EventLoopGroup-1-1" #18 daemon prio=10 os_prio=0 cpu=95914.46ms
elapsed=211826.44s tid=0x00007f9d52141000 nid=0x83 runnable
[0x00007f9d23f6b000]
java.lang.Thread.State: RUNNABLE
at
org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.accept(Native Method)
at
org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.accept(Socket.java:399)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollServerChannel$EpollServerSocketUnsafe.epollInReady(AbstractEpollServerChannel.java:112)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at
org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run([email protected]/Thread.java:829)
{noformat}
The thread is in Runnable state actually? But maybe if it is blocked in a
native method the state will not be changed...
Checked the related code, seems nothing unusual...
We are in processReady method which means epollWait tells us there are
something to read.
{code}
do {
// lastBytesRead represents the fd. We use
lastBytesRead because it must be set so that the
// EpollRecvByteAllocatorHandle knows if it should try
to read again or not when autoRead is
// enabled.
allocHandle.lastBytesRead(socket.accept(acceptedAddress)); // <==== we are
blocked here
if (allocHandle.lastBytesRead() == -1) {
// this means everything was handled for now
break;
}
allocHandle.incMessagesRead(1);
readPending = false;
pipeline.fireChannelRead(newChildChannel(allocHandle.lastBytesRead(),
acceptedAddress, 1,
acceptedAddress[0]));
} while (allocHandle.continueReading());
{code}
The java code
{code}
public final int accept(byte[] addr) throws IOException {
int res = accept(fd, addr);
if (res >= 0) {
return res;
}
if (res == ERRNO_EAGAIN_NEGATIVE || res == ERRNO_EWOULDBLOCK_NEGATIVE) {
// Everything consumed so just return -1 here.
return -1;
}
throw newIOException("accept", res);
}
{code}
The native code
{code}
static jint netty_unix_socket_accept(JNIEnv* env, jclass clazz, jint fd,
jbyteArray acceptedAddress) {
jint socketFd;
jsize len;
jbyte len_b;
int err;
struct sockaddr_storage addr;
socklen_t address_len = sizeof(addr);
for (;;) {
#ifdef SOCK_NONBLOCK
if (accept4) {
socketFd = accept4(fd, (struct sockaddr*) &addr, &address_len,
SOCK_NONBLOCK | SOCK_CLOEXEC);
} else {
#endif
socketFd = accept(fd, (struct sockaddr*) &addr, &address_len);
#ifdef SOCK_NONBLOCK
}
#endif
if (socketFd != -1) {
break;
}
if ((err = errno) != EINTR) {
return -err;
}
}
len = addressLength(&addr);
len_b = (jbyte) len;
// Fill in remote address details
(*env)->SetByteArrayRegion(env, acceptedAddress, 0, 1, (jbyte*) &len_b);
initInetSocketAddressArray(env, &addr, acceptedAddress, 1, len);
if (accept4) {
return socketFd;
}
if (fcntl(socketFd, F_SETFD, FD_CLOEXEC) == -1 || fcntl(socketFd, F_SETFL,
O_NONBLOCK) == -1) {
// accept4 was not present so need two more sys-calls ...
return -errno;
}
return socketFd;
}
{code}
The only strange thing to me is that why we need to call socket.accept
everytime when reading, all other code seems OK. But I'm not an expert of
epoll, I think we can file an issue for netty to ask if they have met similar
problem before.
Meanwhile, you could try to switch back to NIO to see if it could fix the
problem...
Thanks.
> Intra-process client connections cause netty EventLoop deadlock
> ---------------------------------------------------------------
>
> Key: HBASE-28156
> URL: https://issues.apache.org/jira/browse/HBASE-28156
> Project: HBase
> Issue Type: Bug
> Reporter: Bryan Beaudreault
> Priority: Major
> Attachments: hmaster-foo-hb2-a-1-5c7944546c-vdzw4.threads.5
>
>
> We've had a few operational incidents over the past few months where our
> HMaster stops accepting new connections, but can continue processing requests
> from existing ones. Finally I was able to get heap and thread dumps to
> confirm what's happening.
> The core trigger is HBASE-24687, where the MobFileCleanerChore is not using
> ClusterConnection. I've prodded the linked PR to get that resolved and will
> take it over if I don't hear soon.
> In this case, the chore is using the NettyRpcClient to make a local rpc call
> to the same NettyRpcServer in the process. Due to
> [NettyEventLoopGroupConfig|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/NettyEventLoopGroupConfig.java#L98],
> we use the same EventLoopGroup for both the RPC Client and the RPC Server.
> What happens rarely is that the local client for MobFileCleanerChore gets
> assigned to RS-EventLoopGroup-1-1. Since we share the EventLoopGroupConfig,
> and [we don't specify a separate parent
> group|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java#L155],
> that group is also the group which processes new connections.
> What we see in this case is that RS-EventLoopGroup-1-1 gets hung in
> Socket.accept. Since the client side is on the same EventLoop, it's tasks get
> stuck in a queue waiting for the executor. So the client can't send the
> request that the server Socket is waiting for.
> Further, the client/chore gets stuck waiting on BlockingRpcCallback.get(). We
> use an HWT TimerTask to cancel overdue requests, but it only gets scheduled
> [once NettyRpcConnection.sendRequest0 is
> executed|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcConnection.java#L371].
> But sendRequest0 [executes on the
> EventLoop|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcConnection.java#L393],
> and thus gets similarly stuck. So we never schedule a timeout and the chore
> gets stuck forever.
> While fixing HBASE-24687 will fix this case, I think we should improve our
> netty configuration here so we can avoid problems like this if we ever do
> intra-process RPC calls again (there may already be others, not sure).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
