Hi Tom, From the stacktrace it appears as Beryllium SR3 - 1.3.3.Beryllium-SR3 As I recollect, this appears as symptom which called for pre-leader fix. So, would it be better to try out Boron ?
Regards Muthu From: controller-dev-boun...@lists.opendaylight.org [mailto:controller-dev-boun...@lists.opendaylight.org] On Behalf Of Tom Pantelis Sent: Wednesday, March 01, 2017 9:03 PM To: Sivasamy Kaliappan <sivasa...@gmail.com> Cc: controller-dev <controller-dev@lists.opendaylight.org> Subject: Re: [controller-dev] EOS entity without an owner after data store exception Please open a bug. If it's reproducible, it would be helpful to enable debug for org.opendaylight.controller.cluster.datastore.entityownership on each node and provide the log files. On Wed, Mar 1, 2017 at 4:34 AM, Sivasamy Kaliappan <sivasa...@gmail.com<mailto:sivasa...@gmail.com>> wrote: All, We have a 3 node cluster and an entity A defined. On startup all the nodes in cluster creates the entity and register the same. Following is the sequence of events: 1. During cluster startup an owner for the entity A is elected and all listeners are informed 2. A leader for entity ownership shard is elected (different from entity leader) 3. After this election I am seeing below data store exception in the newly elected shard leader 4. After this exception I am getting an owner changed event where hasOwner=false. i.e currently there no owner for this entity and it remains in this state forever 5. org.opendaylight.controller.cluster.datastore.Shard.finishCommit() method has a comment that during edge cases data store will throw IllegalStateException Is this behavior expected? What should we do when there is no owner for the entity in the cluster? No Owner Event: 2017-01-27 22:13:25,920 | INFO | lt-dispatcher-16 | EntityOwnerChangeListener | 283 - com.aaa.odl - 0.1.0.SNAPSHOT | ownershipChanged:request,handleOwnershipChanged: EntityOwnershipChanged [entity=Entity{type='controller', id=/(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)entity/entity[{(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)name=controller}]}, wasOwner=false, isOwner=false, hasOwner=false, inJeopardy=false] event received for entity Entity{type='controller', id=/(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)entity/entity[{(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)name=controller}]} Exception: 2017-01-27 22:13:25,787 | WARN | lt-dispatcher-17 | EntityOwnershipShard | 140 - org.opendaylight.controller.sal-akka-raft - 1.3.3.Beryllium-SR3 | member-2-shard-entity-ownership-operational: commit failed for transaction member-2-txn-2-1485584005749 - retrying as foreign candidate java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82<mailto:org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82> and candidate base org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559<mailto:org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559> differ. at com.google.common.util.concurrent.Futures$ImmediateFailedFuture.get(Futures.java:190)[37:com.google.guava:18.0.0] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.commit(ShardCommitCoordinator.java:670)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:352)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:420)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:668)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:225)[140:org.opendaylight.controller.sal-akka-raft:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[139:org.opendaylight.controller.sal-clustering-commons:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:276)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipShard.onReceiveCommand(EntityOwnershipShard.java:137)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[133:com.typesafe.akka.persistence.experimental:2.3.14] at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[139:org.opendaylight.controller.sal-clustering-commons:1.3.3.Beryllium-SR3] at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[128:com.typesafe.akka.actor:2.3.14] at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[133:com.typesafe.akka.persistence.experimental:2.3.14] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[128:com.typesafe.akka.actor:2.3.14] at akka.actor.ActorCell.invoke(ActorCell.scala:487)[128:com.typesafe.akka.actor:2.3.14] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[128:com.typesafe.akka.actor:2.3.14] at akka.dispatch.Mailbox.run(Mailbox.scala:220)[128:com.typesafe.akka.actor:2.3.14] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[128:com.typesafe.akka.actor:2.3.14] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c] Caused by: java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82<mailto:org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82> and candidate base org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559<mailto:org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559> differ. at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)[57:org.opendaylight.yangtools.yang-data-impl:0.8.3.Beryllium-SR3] at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:106)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3] ... 29 more Thanks _______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org<mailto:controller-dev@lists.opendaylight.org> https://lists.opendaylight.org/mailman/listinfo/controller-dev
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
