My guess is that because of the connection flapping it ends up submitting many FutureCallback for setRole future. And if you don't provide an explicit executor for Futures.addCallback, it will run that FutureCallback on the native thread. So looks like connection flapping is accumulating multiple FutureCallback and they probably are getting cancelled and directExecutor tries to run it through native threads and it doesn't find the thread to run.
Santosh, Can you also paste the output of "ulimit -u" command from the machine where controller crashed? Thanks On Thu, Feb 16, 2017 at 11:54 PM, Muthukumaran K < [email protected]> wrote: > Hi Santosh, > > > > Couple of questions - > > a) Does crash happen instantaneously – I mean within shorter > intervals or degradation is progressive over time ? > > b) since you emulate the flaps, is it possible to control the > frequency of flapping with your emulation ? > > > > If (a) holds true or (b) is possible in your emulation, trend of threads > created can be captured using tools like VisualVM or even from JConsole to > find if there is a progressively increasing thread-creation happening for > this scenario > > > > Btw, I assume you are using only single ODL controller and not a cluster – > right ? > > > > Regards > > Muthu > > > > > > > > > > > > *From:* [email protected] [mailto: > [email protected]] *On Behalf Of *Santosh > Singh > *Sent:* Friday, February 17, 2017 1:08 PM > *To:* Anil Vishnoi <[email protected]> > *Cc:* [email protected] > *Subject:* Re: [openflowplugin-dev] opendaylight [ Lithuim ] crashes due > to "java.lang.OutOfMemoryError: unable to create new native thread" > > > > Hi Anil , > > > > Thanks for your response .. > > > > We are running with following memory option , I think which is > sufficient for ODL instance having 150 OF connection. > > > > -Xms128M -Xmx31393m -XX:MaxPermSize=15696m > > > > Any thoughts on this ?? > > > > We would be trying to recreate this issue in order to get heap dump .... > > > > Thanks > > Santosh > > > > On Fri, Feb 17, 2017 at 12:53 PM, Anil Vishnoi <[email protected]> > wrote: > > Hi Santosh, > > > > Looks like your controller crashed while spawning a new native JVM thread, > because your JVM Is out of native heap space. > > > > Can you increase your native heap space and see if you still hit the issue > (it might take longer to recreate the issue). Meanwhile if you have the > heapdump, please upload the heapdump, that will help in analyzing the > possible cause. > > > > Thanks > > Anil > > > > On Thu, Feb 16, 2017 at 11:05 PM, Santosh Singh <[email protected]> > wrote: > > Hello openflowplugin developers , > > > > I have been using lithium release of opendaylight. We are seeing ODL > crashes with error mentioned in the subject line , when we test the > scenario of frequent connection flap . > > > > If this issue has been already addressed as part of the latest release , > could anyone point to the corresponding bug. > > > > I have pasted the complete stack trace at the below of this mail.. > > > > Thanks > > Santosh > > > > > > > > 2017-02-12 22:19:15,360 | ERROR | lt-dispatcher-27 | ActorSystemImpl | 156 > - com.typesafe.akka.slf4j - 2.3.10 | Uncaught error from thread > [opendaylight-cluster-data-akka.actor.default-dispatcher-4] shutting down > JVM since 'akka.jvm-exit-on-fatal-error' is enabled > java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method)[:1.7.0_95] > at java.lang.Thread.start(Thread.java:714)[:1.7.0_95] > at java.util.concurrent.ThreadPoolExecutor.addWorker( > ThreadPoolExecutor.java:949)[:1.7.0_95] > at java.util.concurrent.ThreadPoolExecutor.execute( > ThreadPoolExecutor.java:1360)[:1.7.0_95] > at java.util.concurrent.Executors$DelegatedExecutorService. > execute(Executors.java:628)[:1.7.0_95] > at com.google.common.util.concurrent.MoreExecutors$ > ListeningDecorator.execute(MoreExecutors.java:550)[51: > com.google.guava:18.0.0] > at java.util.concurrent.AbstractExecutorService.submit( > AbstractExecutorService.java:132)[:1.7.0_95] > at com.google.common.util.concurrent.AbstractListeningExecutorServi > ce.submit(AbstractListeningExecutorService.java:58)[51:com.google. > guava:18.0.0] > at org.opendaylight.openflowplugin.impl.services. > SalRoleServiceImpl.setRole(SalRoleServiceImpl.java:109) > at org.opendaylight.openflowplugin.impl.role. > RoleContextImpl.onRoleChanged(RoleContextImpl.java:110) > at org.opendaylight.openflowplugin.impl.role.OpenflowOwnershipListener. > ownershipChanged(OpenflowOwnershipListener.java:62) > at org.opendaylight.controller.cluster.datastore.entityownership. > EntityOwnershipListenerActor.onEntityOwnershipChanged( > EntityOwnershipListenerActor.java:44)[170:org.opendaylight. > controller.sal-distributed-datastore:1.2.4.SNAPSHOT] > at org.opendaylight.controller.cluster.datastore.entityownership. > EntityOwnershipListenerActor.handleReceive(EntityOwnershipListenerActor. > java:36)[170:org.opendaylight.controller.sal-distributed- > datastore:1.2.4.SNAPSHOT] > at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor. > onReceive(AbstractUntypedActor.java:34)[162:org.opendaylight. > controller.sal-clustering-commons:1.2.4.SNAPSHOT] > at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse( > UntypedActor.scala:167)[155:com.typesafe.akka.actor:2.3.10] > at akka.actor.Actor$class.aroundReceive(Actor.scala:467) > [155:com.typesafe.akka.actor:2.3.10] > > > > _______________________________________________ > openflowplugin-dev mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev > > > > > > -- > > Thanks > > Anil > > > -- Thanks Anil
_______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
