Please open a bug. If it's reproducible, it would be helpful to enable
debug for org.opendaylight.controller.cluster.datastore.entityownership on
each node and provide the log files.
On Wed, Mar 1, 2017 at 4:34 AM, Sivasamy Kaliappan <sivasa...@gmail.com>
wrote:
> All,
>
> We have a 3 node cluster and an entity A defined. On startup all the nodes
> in cluster creates the entity and register the same. Following is the
> sequence of events:
>
> 1. During cluster startup an owner for the entity A is elected and all
> listeners are informed
> 2. A leader for entity ownership shard is elected (different from
> entity leader)
> 3. After this election I am seeing below data store exception in the
> newly elected shard leader
> 4. After this exception I am getting an owner changed event where
> hasOwner=false. i.e currently there no owner for this entity and it remains
> in this state forever
> 5. org.opendaylight.controller.cluster.datastore.Shard.finishCommit()
> method has a comment that during edge cases data store will throw
> IllegalStateException
>
> Is this behavior expected? What should we do when there is no owner for
> the entity in the cluster?
>
> No Owner Event:
>
> 2017-01-27 22:13:25,920 | INFO | lt-dispatcher-16 |
> EntityOwnerChangeListener | 283 - com.aaa.odl - 0.1.0.SNAPSHOT |
> ownershipChanged:request,handleOwnershipChanged: EntityOwnershipChanged
> [entity=Entity{type='controller', id=/(urn:opendaylight:params:
> xml:ns:yang:controller:md:sal:core:general-entity?revision=
> 2015-08-20)entity/entity[{(urn:opendaylight:params:xml:
> ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)name=controller}]},
> wasOwner=false, isOwner=false, hasOwner=false, inJeopardy=false] event
> received for entity Entity{type='controller', id=/(urn:opendaylight:params:
> xml:ns:yang:controller:md:sal:core:general-entity?revision=
> 2015-08-20)entity/entity[{(urn:opendaylight:params:xml:
> ns:yang:controller:md:sal:core:general-entity?revision=
> 2015-08-20)name=controller}]}
>
>
> Exception:
>
> *2017-01-27 22:13:25,787 | WARN | lt-dispatcher-17 | EntityOwnershipShard
> | 140 - org.opendaylight.controller.sal-akka-raft -
> 1.3.3.Beryllium-SR3 | member-2-shard-entity-ownership-operational: commit
> failed for transaction member-2-txn-2-1485584005749 - retrying as foreign
> candidate*
> *java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
> Store tree
> org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82
> and candidate base
> org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559
> differ.*
> * at
> com.google.common.util.concurrent.Futures$ImmediateFailedFuture.get(Futures.java:190)[37:com.google.guava:18.0.0]*
> * at
> org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.commit(ShardCommitCoordinator.java:670)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
> * at
> org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:352)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
> * at
> org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:420)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
> * at
> org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:668)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
> * at
> org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:225)[140:org.opendaylight.controller.sal-akka-raft:1.3.3.Beryllium-SR3]*
> * at
> org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[139:org.opendaylight.controller.sal-clustering-commons:1.3.3.Beryllium-SR3]*
> * at
> org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:276)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
> * at
> org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipShard.onReceiveCommand(EntityOwnershipShard.java:137)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
> * at
> akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
> * at
> org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[139:org.opendaylight.controller.sal-clustering-commons:1.3.3.Beryllium-SR3]*
> * at
> akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[128:com.typesafe.akka.actor:2.3.14]*
> * at
> akka.persistence.Recovery$State$class.process(Recovery.scala:30)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
> * at
> akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
> * at
> akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
> * at
> akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
> * at
> akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
> * at
> akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
> * at
> akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
> * at
> akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
> * at
> akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
> * at
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[128:com.typesafe.akka.actor:2.3.14]*
> * at
> akka.actor.ActorCell.invoke(ActorCell.scala:487)[128:com.typesafe.akka.actor:2.3.14]*
> * at
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[128:com.typesafe.akka.actor:2.3.14]*
> * at
> akka.dispatch.Mailbox.run(Mailbox.scala:220)[128:com.typesafe.akka.actor:2.3.14]*
> * at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[128:com.typesafe.akka.actor:2.3.14]*
> * at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
> * at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
> * at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
> * at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
> *Caused by: java.lang.IllegalStateException: Store tree
> org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82
> and candidate base
> org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559
> differ.*
> * at
> org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)[57:org.opendaylight.yangtools.yang-data-impl:0.8.3.Beryllium-SR3]*
> * at
> org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:106)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
> * ... 29 more*
>
> Thanks
>
> _______________________________________________
> controller-dev mailing list
> controller-dev@lists.opendaylight.org
> https://lists.opendaylight.org/mailman/listinfo/controller-dev
>
>
_______________________________________________
controller-dev mailing list
controller-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/controller-dev
