+openflowplugin-dev & +ovsdb-dev: Tom,
On Mon, May 29, 2017 at 2:57 PM, Tom Pantelis <tompante...@gmail.com> wrote: Thanks a lot for replying, really appreciate it! It looks like the Dispatcher was for data change notifications. I suspect a > listener was hung or responding slowly so the actor's mailbox filled up > with change notifications. I would suggest getting a thread dump next time. > Turn out no need to wait for next time - just figured out that we can obtain thread dumps à posteriori from an HPROF using MAT... see the [4] Bug7370_Threads.zip HTML report just attached to Bug 7370. It shows 604 threads (a lot?), many of which are e.g. parked ForkJoinPool, and a number of them related to ovsdb and openflowplugin stuff... so what are we looking for, in this thread dump? I haven't looked thread each thread's stack yet, but this one vaguely looks like what you may mean by "a listener was hung or responding slowly" (causing "the actor's mailbox filled upwith change notifications"), could it possibly be the reason for / having something to do with this OOM: opendaylight-cluster-data-akka.actor.default-dispatcher-16 at sun.misc.Unsafe.park(ZJ)V (Native Method) at java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()Z (AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(I)V (AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(I)V (AbstractQueuedSynchronizer.java:1304) at com.google.common.util.concurrent.AbstractFuture$Sync.get()Ljava/lang/Object; (AbstractFuture.java:285) at com.google.common.util.concurrent.AbstractFuture.get()Ljava/lang/Object; (AbstractFuture.java:116) at org.opendaylight.openflowplugin.impl.util.DeviceInitializationUtils.initializeNodeInformation(Lorg/opendaylight/openflowplugin/api/openflow/device/DeviceContext;ZLorg/opendaylight/openflowplugin/openflow/md/core/sal/convertor/ConvertorExecutor;)V (DeviceInitializationUtils.java:155) at org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.onContextInstantiateService(Lorg/opendaylight/openflowplugin/api/openflow/connection/ConnectionContext;)Z (DeviceContextImpl.java:730) at org.opendaylight.openflowplugin.impl.lifecycle.LifecycleServiceImpl.instantiateServiceInstance()V (LifecycleServiceImpl.java:53) at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceRegistrationDelegator.instantiateServiceInstance()V (ClusterSingletonServiceRegistrationDelegator.java:46) at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.takeOwnership()V (ClusterSingletonServiceGroupImpl.java:291) at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.ownershipChanged(Lorg/opendaylight/mdsal/eos/common/api/GenericEntityOwnershipChange;)V (ClusterSingletonServiceGroupImpl.java:237) at org.opendaylight.mdsal.singleton.dom.impl.AbstractClusterSingletonServiceProviderImpl.ownershipChanged(Lorg/opendaylight/mdsal/eos/common/api/GenericEntityOwnershipChange;)V (AbstractClusterSingletonServiceProviderImpl.java:145) at org.opendaylight.mdsal.singleton.dom.impl.DOMClusterSingletonServiceProviderImpl.ownershipChanged(Lorg/opendaylight/mdsal/eos/dom/api/DOMEntityOwnershipChange;)V (DOMClusterSingletonServiceProviderImpl.java:23) at org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipListenerActor.onEntityOwnershipChanged(Lorg/opendaylight/mdsal/eos/dom/api/DOMEntityOwnershipChange;)V (EntityOwnershipListenerActor.java:46) at org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipListenerActor.handleReceive(Ljava/lang/Object;)V (EntityOwnershipListenerActor.java:36) at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(Ljava/lang/Object;)V (AbstractUntypedActor.java:26) at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(Ljava/lang/Object;Lscala/Function1;)Ljava/lang/Object; (UntypedActor.scala:165) at akka.actor.Actor$class.aroundReceive(Lakka/actor/Actor;Lscala/PartialFunction;Ljava/lang/Object;)V (Actor.scala:484) at akka.actor.UntypedActor.aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V (UntypedActor.scala:95) at akka.actor.ActorCell.receiveMessage(Ljava/lang/Object;)V (ActorCell.scala:526) at akka.actor.ActorCell.invoke(Lakka/dispatch/Envelope;)V (ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(IJ)V (Mailbox.scala:257) at akka.dispatch.Mailbox.run()V (Mailbox.scala:224) at akka.dispatch.Mailbox.exec()Z (Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec()I (ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(Lscala/concurrent/forkjoin/ForkJoinTask;)V (ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(Lscala/concurrent/forkjoin/ForkJoinPool$WorkQueue;)V (ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run()V (ForkJoinWorkerThread.java:107) > On Mon, May 29, 2017 at 7:52 AM, Michael Vorburger <vorbur...@redhat.com> > wrote: > >> Hi guys, >> >> I just ran MAT([1]) over an HPROF heap dump on OOM in Bug 7370, and it >> (MAT) raises a "leak suspect" in akka.dispatch.Dispatcher - see the [3] >> java_pid19570_Leak_Suspects.zip just attached to Bug 7370 ... questions: >> >> Is this perhaps something you jump at with an "ah that, we know about it >> and already fixed that in ..." ? >> >> If not, how do we go about better understanding the root cause of this, >> and be able to eventually fix this? >> >> My underlying assumption here is that isn't "normal" and not just "by >> design" - if it is, I'd love some education... like I'm hoping that the >> conclusion here isn't simply that MD SAL's data store is a dumb in-memory >> data base which basically just takes a huge amount of GBs to keep (all) >> YANG model instances on the heap - or is it? >> >> Tx, >> M. >> >> [1] https://www.eclipse.org/mat/ >> >> [2] https://bugs.opendaylight.org/show_bug.cgi?id=7370 >> >> [3] https://bugs.opendaylight.org/attachment.cgi?id=1816 >> > [4] https://bugs.opendaylight.org/attachment.cgi?id=1819 > -- >> Michael Vorburger, Red Hat >> vorbur...@redhat.com | IRC: vorburger @freenode | ~ = http://vorburger.ch >> >> _______________________________________________ >> mdsal-dev mailing list >> mdsal-...@lists.opendaylight.org >> https://lists.opendaylight.org/mailman/listinfo/mdsal-dev >> >> >
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
