Victoria Markman created DRILL-3842:
---------------------------------------
Summary: JVM dies if drill attempts to read too many files in the
directory that blow up heap
Key: DRILL-3842
URL: https://issues.apache.org/jira/browse/DRILL-3842
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.1.0, 1.2.0
Reporter: Victoria Markman
Priority: Critical
Run {{select count(*) from t1}} where t1 directory consists of 1.9 million
little parquet files. The outcome: drillbit is dead and out of working set.
1. Client never got response back from the server
2. drillbit.log
{code}
2015-09-25 17:56:55,935 [29fa756f-894d-0340-3661-b925bff0fe11:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Took 47999 ms to get file statuses
2015-09-25 18:43:19,871 [BitServer-4] INFO
o.a.d.exec.rpc.control.ControlServer - RPC connection /10.10.88.135:31011 <-->
/10.10.88.135:51675 (control server) timed out. Timeout was set to 300
seconds. Closing connection.
2015-09-25 18:50:06,026 [BitServer-3] INFO
o.a.d.exec.rpc.control.ControlClient - Channel closed /10.10.88.135:51675 <-->
/10.10.88.135:31011.
2015-09-25 18:50:06,032 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler
- Exception in RPC communication. Connection: /10.10.88.135:31010 <-->
/10.10.88.133:51612 (user client). Closing connection.
java.lang.OutOfMemoryError: Java heap space
{code}
drillbit.out
{code}
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "main-SendThread(atsqa4-133.qa.lab:5181)"
Exception in thread "WorkManager.StatusThread" java.lang.OutOfMemoryError: Java
heap space
2015-09-25 18:53:52
Full thread dump OpenJDK 64-Bit Server VM (24.65-b04 mixed mode):
{code}
jstack
{code}
[Fri Sep 25 18:53:29 ] # jstack 63205
63205: Unable to open socket file: target process not responding or HotSpot VM
not loaded
The -F option can be used when the target process is not responding
{code}
jstack -F
{code}
Attaching to process ID 63205, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.65-b04
java.lang.RuntimeException: Unable to deduce type of thread from address
0x0000000004093800 (expected type JavaThread, CompilerThread, ServiceThread,
JvmtiAgentThread, or SurrogateLockerThread)
at
sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
at
sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
at
sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
at
sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type
of address 0x0000000004093800
at
sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
at
sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
at
sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
... 15 more
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Caused by: java.lang.RuntimeException: Unable to deduce type of thread from
address 0x0000000004093800 (expected type JavaThread, CompilerThread,
ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
at
sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
... 6 more
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type
of address 0x0000000004093800
at
sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
at
sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
at
sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
... 12 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
