Quanlong Huang created IMPALA-10842:
---------------------------------------
Summary: Show full stacktraces of jvm threads in web UI
Key: IMPALA-10842
URL: https://issues.apache.org/jira/browse/IMPALA-10842
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Quanlong Huang
Attachments: image-2021-08-05-11-01-20-978.png
IMPALA-4885 adds a web page showing all the JVM threads, which saves us some
time to ssh to the machine and perform jstack manually.
However, the summary column doesn't show the full stacktrace if it has more
than 8 frames. For instance, when debugging a long running DDL in catalogd, I
search for "execDdl" in the search box and get nothing since "execDdl" is
truncated:
!image-2021-08-05-11-01-20-978.png|width=905,height=208!
I finally did a jstack and find the thread:
{code:java}
"Thread-11" #42 prio=5 os_prio=0 tid=0x000000000f86c000 nid=0x4c99 runnable
[0x00007f9a8f8e1000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000005d4dbfa88> (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_objects_by_name_req(ThriftHiveMetastore.java:2492)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_objects_by_name_req(ThriftHiveMetastore.java:2479)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:2333)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableObjectsByName(HiveMetaStoreClient.java:2314)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
at com.sun.proxy.$Proxy11.getTableObjectsByName(Unknown Source)
at
org.apache.impala.service.CatalogOpExecutor.dropTablesFromKudu(CatalogOpExecutor.java:2349)
at
org.apache.impala.service.CatalogOpExecutor.dropDatabase(CatalogOpExecutor.java:2255)
- locked <0x00000005d4a101e8> (a java.lang.Object)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:462)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:240)
{code}
The cause is that we simply use TheadInfo#toString() which truncates the
stacktrace:
https://github.com/apache/impala/blob/47d868fe586905b2f4ea347a97ce582a45aac405/fe/src/main/java/org/apache/impala/common/JniUtil.java#L281
We can improve this to get the whole stacktrace.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
