Eugene Chung created HIVE-23164:
-----------------------------------
Summary: server is not properly terminated because of non-daemon
threads
Key: HIVE-23164
URL: https://issues.apache.org/jira/browse/HIVE-23164
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: Eugene Chung
Attachments: thread_dump_hiveserver2_is_not_terminated.txt
As you know, HiveServer2 which receives the deregister command is at first
preparing shutdown. If there're no remaining sessions, HiveServer2.stop() is
called to shut down. But I found the case that the HiveServer2 JVM is not
terminated even if HiveServer2.stop() is called and properly processed. The
case is always occurred when the local(embedded) metastore is used.
I've attached the full thread dump describing the situation.
[^thread_dump_hiveserver2_is_not_terminated.txt]
In this thread dump, you can see some bunch of 'daemon' threads, NO main
thread, and some non-daemon thread(or user thread)s. As specified by
[https://www.baeldung.com/java-daemon-thread], if there exists at least one
user thread which is alive, JVM does not terminate. (Note that DestroyJavaVM
thread is non-daemon but it's special.)
{code:java}
"pool-8-thread-1" #24 prio=5 os_prio=0 tid=0x00007f52ad1fc000 nid=0x821c
waiting on condition [0x00007f525c500000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003cfa057c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
{code}
The thread above is created by ScheduledThreadPoolExecutor(int coreSize) with
default ThreadFactory which always creates non-daemon thread. If such thread
pool is not shut down with ScheduledThreadPoolExecutor.shutdown() method, JVM
cannot terminate! The only way to kill is TERM signal. If JVM receives TERM
signal, it ignores non-daemon threads.
So I have been digging modules which create ScheduledThreadPoolExecutor with
non-daemon threads and now I got it. As you may guess, it's local(embedded)
metastore. It's created by
org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler#startAlwaysTaskThreads()
and ScheduledThreadPoolExecutor.shutdown() is never called.
Plus, I found another usage of creating such ScheduledThreadPoolExecutor and
not calling its shutdown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
