Based on the current observation , thread count is increasing with every flap. So over the period of time , based on system spec and number of devices connection flapping, it would again crash .
I agree with Anil explanations, but this seems to be current design limitations. Adding lookup before proceeding for creating new future for the same device could address this limitation. Any thoughts on this or if something similar exists in recent release. On 17-Feb-2017 1:56 pm, "Santosh Singh" <[email protected]> wrote: > > ulimit -u : 127973 > > > > On Fri, Feb 17, 2017 at 1:36 PM, Santosh Singh <[email protected]> > wrote: > >> Hi Muthukumaran , >> >> You are right it is standalone deployment of ODL . >> >> So basically we are emulating flap every 30 second . >> >> We are trying to recreate this issue , we are keeping track of the thread >> count post every flap . >> >> overall thread count is increasing irrespective intermediate decrease in >> the thread count. >> >> Thanks >> Santosh >> >> On Fri, Feb 17, 2017 at 1:24 PM, Muthukumaran K < >> [email protected]> wrote: >> >>> Hi Santosh, >>> >>> >>> >>> Couple of questions - >>> >>> a) Does crash happen instantaneously – I mean within shorter >>> intervals or degradation is progressive over time ? >>> >>> b) since you emulate the flaps, is it possible to control the >>> frequency of flapping with your emulation ? >>> >>> >>> >>> If (a) holds true or (b) is possible in your emulation, trend of threads >>> created can be captured using tools like VisualVM or even from JConsole to >>> find if there is a progressively increasing thread-creation happening for >>> this scenario >>> >>> >>> >>> Btw, I assume you are using only single ODL controller and not a cluster >>> – right ? >>> >>> >>> >>> Regards >>> >>> Muthu >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *From:* [email protected] [mailto: >>> [email protected]] *On Behalf Of *Santosh >>> Singh >>> *Sent:* Friday, February 17, 2017 1:08 PM >>> *To:* Anil Vishnoi <[email protected]> >>> *Cc:* [email protected] >>> *Subject:* Re: [openflowplugin-dev] opendaylight [ Lithuim ] crashes >>> due to "java.lang.OutOfMemoryError: unable to create new native thread" >>> >>> >>> >>> Hi Anil , >>> >>> >>> >>> Thanks for your response .. >>> >>> >>> >>> We are running with following memory option , I think which is >>> sufficient for ODL instance having 150 OF connection. >>> >>> >>> >>> -Xms128M -Xmx31393m -XX:MaxPermSize=15696m >>> >>> >>> >>> Any thoughts on this ?? >>> >>> >>> >>> We would be trying to recreate this issue in order to get heap dump .... >>> >>> >>> >>> Thanks >>> >>> Santosh >>> >>> >>> >>> On Fri, Feb 17, 2017 at 12:53 PM, Anil Vishnoi <[email protected]> >>> wrote: >>> >>> Hi Santosh, >>> >>> >>> >>> Looks like your controller crashed while spawning a new native JVM >>> thread, because your JVM Is out of native heap space. >>> >>> >>> >>> Can you increase your native heap space and see if you still hit the >>> issue (it might take longer to recreate the issue). Meanwhile if you have >>> the heapdump, please upload the heapdump, that will help in analyzing the >>> possible cause. >>> >>> >>> >>> Thanks >>> >>> Anil >>> >>> >>> >>> On Thu, Feb 16, 2017 at 11:05 PM, Santosh Singh < >>> [email protected]> wrote: >>> >>> Hello openflowplugin developers , >>> >>> >>> >>> I have been using lithium release of opendaylight. We are seeing ODL >>> crashes with error mentioned in the subject line , when we test the >>> scenario of frequent connection flap . >>> >>> >>> >>> If this issue has been already addressed as part of the latest release >>> , could anyone point to the corresponding bug. >>> >>> >>> >>> I have pasted the complete stack trace at the below of this mail.. >>> >>> >>> >>> Thanks >>> >>> Santosh >>> >>> >>> >>> >>> >>> >>> >>> 2017-02-12 22:19:15,360 | ERROR | lt-dispatcher-27 | ActorSystemImpl | >>> 156 - com.typesafe.akka.slf4j - 2.3.10 | Uncaught error from thread >>> [opendaylight-cluster-data-akka.actor.default-dispatcher-4] shutting >>> down JVM since 'akka.jvm-exit-on-fatal-error' is enabled >>> java.lang.OutOfMemoryError: unable to create new native thread >>> at java.lang.Thread.start0(Native Method)[:1.7.0_95] >>> at java.lang.Thread.start(Thread.java:714)[:1.7.0_95] >>> at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPool >>> Executor.java:949)[:1.7.0_95] >>> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolEx >>> ecutor.java:1360)[:1.7.0_95] >>> at java.util.concurrent.Executors$DelegatedExecutorService.exec >>> ute(Executors.java:628)[:1.7.0_95] >>> at com.google.common.util.concurrent.MoreExecutors$ListeningDec >>> orator.execute(MoreExecutors.java:550)[51:com.google.guava:18.0.0] >>> at java.util.concurrent.AbstractExecutorService.submit(Abstract >>> ExecutorService.java:132)[:1.7.0_95] >>> at com.google.common.util.concurrent.AbstractListeningExecutorS >>> ervice.submit(AbstractListeningExecutorService.java:58)[51:c >>> om.google.guava:18.0.0] >>> at org.opendaylight.openflowplugin.impl.services.SalRoleService >>> Impl.setRole(SalRoleServiceImpl.java:109) >>> at org.opendaylight.openflowplugin.impl.role.RoleContextImpl.on >>> RoleChanged(RoleContextImpl.java:110) >>> at org.opendaylight.openflowplugin.impl.role.OpenflowOwnershipL >>> istener.ownershipChanged(OpenflowOwnershipListener.java:62) >>> at org.opendaylight.controller.cluster.datastore.entityownershi >>> p.EntityOwnershipListenerActor.onEntityOwnershipChanged(Enti >>> tyOwnershipListenerActor.java:44)[170:org.opendaylight.contr >>> oller.sal-distributed-datastore:1.2.4.SNAPSHOT] >>> at org.opendaylight.controller.cluster.datastore.entityownershi >>> p.EntityOwnershipListenerActor.handleReceive(EntityOwnership >>> ListenerActor.java:36)[170:org.opendaylight.controller.sal- >>> distributed-datastore:1.2.4.SNAPSHOT] >>> at org.opendaylight.controller.cluster.common.actor.AbstractUnt >>> ypedActor.onReceive(AbstractUntypedActor.java:34)[162:org.op >>> endaylight.controller.sal-clustering-commons:1.2.4.SNAPSHOT] >>> at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(Untyp >>> edActor.scala:167)[155:com.typesafe.akka.actor:2.3.10] >>> at akka.actor.Actor$class.aroundReceive(Actor.scala:467)[155:co >>> m.typesafe.akka.actor:2.3.10] >>> >>> >>> >>> _______________________________________________ >>> openflowplugin-dev mailing list >>> [email protected] >>> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev >>> >>> >>> >>> >>> >>> -- >>> >>> Thanks >>> >>> Anil >>> >>> >>> >> >> >
_______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
