liuguanghua created HDFS-18035:
----------------------------------

             Summary: Interrupt the IPC parameter sending thread when a 
connection is marked closed
                 Key: HDFS-18035
                 URL: https://issues.apache.org/jira/browse/HDFS-18035
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: common
            Reporter: liuguanghua
            Assignee: liuguanghua


A production DFSRouter died with a native OOM while creating a new thread:

--------------- T H R E A D ---------------

Current thread (0x00007ee90de5d800): JavaThread "IPC Client (1604275183) 
connection to yj-hdfsmaster6-001/10.104.9.243:8020 from platform_rd_two" daemon 
[_thread_new, id=65241, stack(0x00007ee1e9442000,0x00007ee1e9543000)]

Stack: [0x00007ee1e9442000,0x00007ee1e9543000], sp=0x00007ee1e9541a20, free 
space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xad21aa] VMError::report_and_die()+0x2ba
V [libjvm.so+0x5022db] report_vm_out_of_memory(char const*, int, unsigned long, 
VMErrorType, char const*)+0x8b
V [libjvm.so+0x92fe33] os::Linux::commit_memory_impl(char*, unsigned long, 
bool)+0x123
V [libjvm.so+0x92fefc] os::pd_commit_memory(char*, unsigned long, bool)+0xc
V [libjvm.so+0x927f8a] os::commit_memory(char*, unsigned long, bool)+0x2a
V [libjvm.so+0x92bc4f] os::pd_create_stack_guard_pages(char*, unsigned 
long)+0x7f
V [libjvm.so+0xa771ce] JavaThread::create_stack_guard_pages()+0x5e
V [libjvm.so+0xa81454] JavaThread::run()+0x34
V [libjvm.so+0x930198] java_start(Thread*)+0x108
C [libpthread.so.0+0x7ea5] start_thread+0xc5

 


The failure was in create_stack_guard_pages() — the JVM could not mmap the 
stack for yet another thread. The thread dump shows why:
||Thread kind||Count||
|{{IPC Parameter Sending Thread for ...}} (blocked)|*25,191*|
|{{IPC Client (...) connection to ...}}|2,319|
|Total Java threads|32,449|

 

Almost all threads target the same subcluster NameNode from the router. The 
sending threads outnumber the actual connections by ~11:1, each one holding a 
~1MB native stack — enough to exhaust the process's address space / 
vm.max_map_count.

Root cause

When an IPC Connection is marked closed (idle timeout, I/O error, or a FATAL 
response), its RpcRequestSender thread (IPC Parameter Sending Thread for ...) 
is left blocked in rpcRequestQueue.poll(maxIdleTime). It only notices the close 
after the poll times out, so the thread — and its stack / mapped memory — 
lingers for up to ipc.client.connection.maxidletime after the connection is 
already gone.

markClosed() calls notifyAll(), which only wakes the Connection.run() thread 
(blocked in waitForWork()'s wait(timeout)). The sender thread blocks inside 
SynchronousQueue.poll(timeout), which does not wait on the connection monitor, 
so it is never woken and only exits once the poll times out. The connection 
thread exits on close; the sender thread stays behind and accumulates. Under 
heavy connection churn (e.g. a DFSRouter recycling pooled connections to its 
subcluster NameNodes), these lingering threads pile up until the JVM can no 
longer create new threads.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to