Hi Janardhan, to sure what's going wrong here. Maybe Till (in CC) has an idea?
Best, Fabian 2016-09-19 19:45 GMT+02:00 Janardhan Reddy <janardhan.re...@olacabs.com>: > HI, > > I cancelled a restarting job from flink UI and the job is stuck in > cancelling state. (Fixed delay restart strategy was configured for the > job). The following error message is present in taskmanager logs. > > akka.remote.OversizedPayloadException: Discarding oversized payload sent > to Actor[akka.tcp://flink@10.200.7.245:42589/user/jobmanager#-146176374]: > max allowed size 10485760 bytes, actual size of encoded class > org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMessage > was 20670224 bytes. > > > Does the leader session message here denote the job cancel message which > is sent to the job manager decorated with leader session id ? > > Thread dump of taskmanager: > > > Attaching to process ID 28948, please wait... > > Debugger attached successfully. > > Server compiler detected. > > JVM version is 25.101-b13 > > Deadlock Detection: > > > java.lang.RuntimeException: Unable to deduce type of thread from address > 0x00007f96e56c9800 (expected type JavaThread, CompilerThread, > ServiceThread, JvmtiAgentThread, or SurrogateLockerThread) > > at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper( > Threads.java:169) > > at sun.jvm.hotspot.runtime.Threads.first(Threads.java:153) > > at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable( > DeadlockDetector.java:149) > > at sun.jvm.hotspot.runtime.DeadlockDetector.print( > DeadlockDetector.java:56) > > at sun.jvm.hotspot.runtime.DeadlockDetector.print( > DeadlockDetector.java:39) > > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:62) > > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) > > at sun.jvm.hotspot.tools.JStack.run(JStack.java:66) > > at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260) > > at sun.jvm.hotspot.tools.Tool.start(Tool.java:223) > > at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) > > at sun.jvm.hotspot.tools.JStack.main(JStack.java:92) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at sun.tools.jstack.JStack.runJStackTool(JStack.java:140) > > at sun.tools.jstack.JStack.main(JStack.java:106) > > Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match > for type of address 0x00007f96e56c9800 > > at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException( > InstanceConstructor.java:62) > > at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor( > VirtualConstructor.java:80) > > at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper( > Threads.java:165) > > ... 17 more > > Can't print deadlocks:Unable to deduce type of thread from address > 0x00007f96e56c9800 (expected type JavaThread, CompilerThread, > ServiceThread, JvmtiAgentThread, or SurrogateLockerThread) > > Exception in thread "main" java.lang.reflect.InvocationTargetException > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at sun.tools.jstack.JStack.runJStackTool(JStack.java:140) > > at sun.tools.jstack.JStack.main(JStack.java:106) > > Caused by: java.lang.RuntimeException: Unable to deduce type of thread > from address 0x00007f96e56c9800 (expected type JavaThread, CompilerThread, > ServiceThread, JvmtiAgentThread, or SurrogateLockerThread) > > at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper( > Threads.java:169) > > at sun.jvm.hotspot.runtime.Threads.first(Threads.java:153) > > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:75) > > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) > > at sun.jvm.hotspot.tools.JStack.run(JStack.java:66) > > at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260) > > at sun.jvm.hotspot.tools.Tool.start(Tool.java:223) > > at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) > > at sun.jvm.hotspot.tools.JStack.main(JStack.java:92) > > ... 6 more > > Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match > for type of address 0x00007f96e56c9800 > > at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException( > InstanceConstructor.java:62) > > at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor( > VirtualConstructor.java:80) > > at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper( > Threads.java:165) > > ... 14 more >
