All,
We have a 3 node cluster and an entity A defined. On startup all the nodes
in cluster creates the entity and register the same. Following is the
sequence of events:
1. During cluster startup an owner for the entity A is elected and all
listeners are informed
2. A leader for entity ownership shard is elected (different from entity
leader)
3. After this election I am seeing below data store exception in the
newly elected shard leader
4. After this exception I am getting an owner changed event where
hasOwner=false. i.e currently there no owner for this entity and it remains
in this state forever
5. org.opendaylight.controller.cluster.datastore.Shard.finishCommit()
method has a comment that during edge cases data store will throw
IllegalStateException
Is this behavior expected? What should we do when there is no owner for the
entity in the cluster?
No Owner Event:
2017-01-27 22:13:25,920 | INFO | lt-dispatcher-16 |
EntityOwnerChangeListener | 283 - com.aaa.odl - 0.1.0.SNAPSHOT |
ownershipChanged:request,handleOwnershipChanged: EntityOwnershipChanged
[entity=Entity{type='controller',
id=/(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)entity/entity[{(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)name=controller}]},
wasOwner=false, isOwner=false, hasOwner=false, inJeopardy=false] event
received for entity Entity{type='controller',
id=/(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)entity/entity[{(urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:general-entity?revision=2015-08-20)name=controller}]}
Exception:
*2017-01-27 22:13:25,787 | WARN | lt-dispatcher-17 | EntityOwnershipShard
| 140 - org.opendaylight.controller.sal-akka-raft -
1.3.3.Beryllium-SR3 | member-2-shard-entity-ownership-operational: commit
failed for transaction member-2-txn-2-1485584005749 - retrying as foreign
candidate*
*java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
Store tree
org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82
and candidate base
org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559
differ.*
* at
com.google.common.util.concurrent.Futures$ImmediateFailedFuture.get(Futures.java:190)[37:com.google.guava:18.0.0]*
* at
org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.commit(ShardCommitCoordinator.java:670)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
* at
org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:352)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
* at
org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:420)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
* at
org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:668)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
* at
org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:225)[140:org.opendaylight.controller.sal-akka-raft:1.3.3.Beryllium-SR3]*
* at
org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[139:org.opendaylight.controller.sal-clustering-commons:1.3.3.Beryllium-SR3]*
* at
org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:276)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
* at
org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipShard.onReceiveCommand(EntityOwnershipShard.java:137)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
* at
akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
* at
org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[139:org.opendaylight.controller.sal-clustering-commons:1.3.3.Beryllium-SR3]*
* at
akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[128:com.typesafe.akka.actor:2.3.14]*
* at
akka.persistence.Recovery$State$class.process(Recovery.scala:30)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
* at
akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
* at
akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
* at
akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
* at
akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
* at
akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
* at
akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
* at
akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
* at
akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
* at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[128:com.typesafe.akka.actor:2.3.14]*
* at
akka.actor.ActorCell.invoke(ActorCell.scala:487)[128:com.typesafe.akka.actor:2.3.14]*
* at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[128:com.typesafe.akka.actor:2.3.14]*
* at
akka.dispatch.Mailbox.run(Mailbox.scala:220)[128:com.typesafe.akka.actor:2.3.14]*
* at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[128:com.typesafe.akka.actor:2.3.14]*
* at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
* at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
* at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
* at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
*Caused by: java.lang.IllegalStateException: Store tree
org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82
and candidate base
org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559
differ.*
* at
org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)[57:org.opendaylight.yangtools.yang-data-impl:0.8.3.Beryllium-SR3]*
* at
org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:106)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
* ... 29 more*
Thanks
_______________________________________________
controller-dev mailing list
controller-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/controller-dev
