Thanks for the update Siva Regards Muthu
From: Sivasamy Kaliappan [mailto:sivasa...@gmail.com] Sent: Wednesday, March 22, 2017 6:03 PM To: Tom Pantelis Cc: Muthukumaran K; controller-dev Subject: Re: [controller-dev] EOS entity without an owner after data store exception We are not seeing this issue after upgrading to Boron. Thanks! On Thu, Mar 2, 2017 at 3:59 PM, Tom Pantelis <tompante...@gmail.com<mailto:tompante...@gmail.com>> wrote: yes - I didn't realize it was Beryllium. I would suggest trying Boron - PreLeader fix as you mentioned plus a lot of hardening in EOS. On Wed, Mar 1, 2017 at 11:35 PM, Muthukumaran K <muthukumara...@ericsson.com<mailto:muthukumara...@ericsson.com>> wrote: Hi Tom, From the stacktrace it appears as Beryllium SR3 - 1.3.3.Beryllium-SR3 As I recollect, this appears as symptom which called for pre-leader fix. So, would it be better to try out Boron ? Regards Muthu From: controller-dev-boun...@lists.opendaylight.org<mailto:controller-dev-boun...@lists.opendaylight.org> [mailto:controller-dev-boun...@lists.opendaylight.org<mailto:controller-dev-boun...@lists.opendaylight.org>] On Behalf Of Tom Pantelis Sent: Wednesday, March 01, 2017 9:03 PM To: Sivasamy Kaliappan <sivasa...@gmail.com<mailto:sivasa...@gmail.com>> Cc: controller-dev <controller-dev@lists.opendaylight.org<mailto:controller-dev@lists.opendaylight.org>> Subject: Re: [controller-dev] EOS entity without an owner after data store exception Please open a bug. If it's reproducible, it would be helpful to enable debug for org.opendaylight.controller.cluster.datastore.entityownership on each node and provide the log files. On Wed, Mar 1, 2017 at 4:34 AM, Sivasamy Kaliappan <sivasa...@gmail.com<mailto:sivasa...@gmail.com>> wrote: All, We have a 3 node cluster and an entity A defined. On startup all the nodes in cluster creates the entity and register the same. Following is the sequence of events: 1. During cluster startup an owner for the entity A is elected and all listeners are informed 2. A leader for entity ownership shard is elected (different from entity leader) 3. After this election I am seeing below data store exception in the newly elected shard leader 4. After this exception I am getting an owner changed event where hasOwner=false. i.e currently there no owner for this entity and it remains in this state forever 5. org.opendaylight.controller.cl<http://org.opendaylight.controller.cl>uster.datastore.Shard.finishCommit() method has a comment that during edge cases data store will throw IllegalStateException Is this behavior expected? What should we do when there is no owner for the entity in the cluster? No Owner Event: 2017-01-27 22:13:25,920 | INFO | lt-dispatcher-16 | EntityOwnerChangeListener | 283 - com.aaa.odl - 0.1.0.SNAPSHOT | ownershipChanged:request,handleOwnershipChanged: EntityOwnershipChanged [entity=Entity{type='controller', id=/(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)entity/entity[{(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)name=controller}]}, wasOwner=false, isOwner=false, hasOwner=false, inJeopardy=false] event received for entity Entity{type='controller', id=/(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)entity/entity[{(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)name=controller}]} Exception: 2017-01-27 22:13:25,787 | WARN | lt-dispatcher-17 | EntityOwnershipShard | 140 - org.opendaylight.controller.sa<http://org.opendaylight.controller.sa>l-akka-raft - 1.3.3.Beryllium-SR3 | member-2-shard-entity-ownership-operational: commit failed for transaction member-2-txn-2-1485584005749 - retrying as foreign candidate java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82<mailto:org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82> and candidate base org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559<mailto:org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559> differ. at com.google.common.util.concurrent.Futures$ImmediateFailedFuture.get(Futures.java:190)[37:com.google.guava:18.0.0] at org.opendaylight.controller.cl<http://org.opendaylight.controller.cl>uster.datastore.ShardCommitCoordinator$CohortEntry.commit(ShardCommitCoordinator.java:670)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cl<http://org.opendaylight.controller.cl>uster.datastore.Shard.finishCommit(Shard.java:352)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cl<http://org.opendaylight.controller.cl>uster.datastore.Shard.finishCommit(Shard.java:420)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cl<http://org.opendaylight.controller.cl>uster.datastore.Shard.applyState(Shard.java:668)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cl<http://org.opendaylight.controller.cl>uster.raft.RaftActor.handleCommand(RaftActor.java:225)[140:org.opendaylight.controller.sal-akka-raft:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cl<http://org.opendaylight.controller.cl>uster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[139:org.opendaylight.controller.sal-clustering-commons:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cl<http://org.opendaylight.controller.cl>uster.datastore.Shard.onReceiveCommand(Shard.java:276)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cl<http://org.opendaylight.controller.cl>uster.datastore.entityownership.EntityOwnershipShard.onReceiveCommand(EntityOwnershipShard.java:137)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[133:com.typesafe.akka.persistence.experimental:2.3.14] at org.opendaylight.controller.cl<http://org.opendaylight.controller.cl>uster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[139:org.opendaylight.controller.sal-clustering-commons:1.3.3.Beryllium-SR3] at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[128:com.typesafe.akka.actor:2.3.14] at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[128:com.typesafe.akka.actor:2.3.14] at akka.actor.ActorCell.invoke(ActorCell.scala:487)[128:com.typesafe.akka.actor:2.3.14] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[128:com.typesafe.akka.actor:2.3.14] at akka.dispatch.Mailbox.run(Mailbox.scala:220)[128:com.typesafe.akka.actor:2.3.14] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[128:com.typesafe.akka.actor:2.3.14] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c] Caused by: java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82<mailto:org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82> and candidate base org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559<mailto:org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559> differ. at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)[57:org.opendaylight.yangtools.yang-data-impl:0.8.3.Beryllium-SR3] at org.opendaylight.controller.cl<http://org.opendaylight.controller.cl>uster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:106)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] ... 29 more Thanks _______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org<mailto:controller-dev@lists.opendaylight.org> https://lists.opendaylight.org/mailman/listinfo/controller-dev -- Regards, Siva.
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
