Hi, Anonymous and inner classes have link to outer class object and might bring it to marshaller. When you set it inner static or separate class you're explicitly saying that you don't need such links.
In thread dumps you need to lookup for waiting or blocked threads. In your case in service node you may find that service thread is waiting on invoke(): "svc-#70" #102 prio=5 os_prio=0 tid=0x00007fe820024800 nid=0x2c44 waiting on condition [0x00007fe7d51f4000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.invoke(GridDhtAtomicCache.java:785) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.invoke(IgniteCacheProxyImpl.java:1338) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.invoke(GatewayProtectedCacheProxy.java:1320) at com.mediaiq.caps.platform.choreography.service.IgniteWorkflowServiceImpl.startWorkflow(IgniteWorkflowServiceImpl.java:165) ... Cache operations are invoked on data nodes, so you may go to data node and find: "sys-stripe-5-#6" #15 prio=5 os_prio=0 tid=0x00007fd96459b800 nid=0x29a7 waiting on condition [0x00007fd94cf9c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4512) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4493) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1326) at org.apache.ignite.internal.processors.cache.GridCacheProxyImpl.get(GridCacheProxyImpl.java:329) at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getCollection(DataStructuresProcessor.java:1001) at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.queue(DataStructuresProcessor.java:794) at org.apache.ignite.internal.processors.datastructures.GridCacheQueueProxy.readResolve(GridCacheQueueProxy.java:495) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:549) at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:917) at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) ... Here OptimizedMarshaller tries to deserialize EntryProcessor, but hanged on deserializing GridCacheQueueProxy aka IgniteQueue. Obviously you do not need to marshal/unmarshal it, and the best solution here would be to overcome it's serialization - remove it from anonymous EntryProcessor context. Thanks! -Dmitry -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/