We are not seeing this issue after upgrading to Boron. Thanks!
On Thu, Mar 2, 2017 at 3:59 PM, Tom Pantelis <tompante...@gmail.com> wrote:
> yes - I didn't realize it was Beryllium. I would suggest trying Boron -
> PreLeader fix as you mentioned plus a lot of hardening in EOS.
>
> On Wed, Mar 1, 2017 at 11:35 PM, Muthukumaran K <
> muthukumara...@ericsson.com> wrote:
>
>> Hi Tom,
>>
>>
>>
>> From the stacktrace it appears as Beryllium SR3 - *1.3.3.Beryllium-SR3 *
>>
>> As I recollect, this appears as symptom which called for pre-leader fix.
>> So, would it be better to try out Boron ?
>>
>>
>>
>> Regards
>>
>> Muthu
>>
>>
>>
>>
>>
>> *From:* controller-dev-boun...@lists.opendaylight.org [mailto:
>> controller-dev-boun...@lists.opendaylight.org] *On Behalf Of *Tom
>> Pantelis
>> *Sent:* Wednesday, March 01, 2017 9:03 PM
>> *To:* Sivasamy Kaliappan <sivasa...@gmail.com>
>> *Cc:* controller-dev <controller-dev@lists.opendaylight.org>
>> *Subject:* Re: [controller-dev] EOS entity without an owner after data
>> store exception
>>
>>
>>
>> Please open a bug. If it's reproducible, it would be helpful to enable
>> debug for org.opendaylight.controller.cluster.datastore.entityownership
>> on each node and provide the log files.
>>
>>
>>
>> On Wed, Mar 1, 2017 at 4:34 AM, Sivasamy Kaliappan <sivasa...@gmail.com>
>> wrote:
>>
>> All,
>>
>>
>>
>> We have a 3 node cluster and an entity A defined. On startup all the
>> nodes in cluster creates the entity and register the same. Following is the
>> sequence of events:
>>
>> 1. During cluster startup an owner for the entity A is elected and
>> all listeners are informed
>> 2. A leader for entity ownership shard is elected (different from
>> entity leader)
>> 3. After this election I am seeing below data store exception in the
>> newly elected shard leader
>> 4. After this exception I am getting an owner changed event where
>> hasOwner=false. i.e currently there no owner for this entity and it
>> remains
>> in this state forever
>> 5. org.opendaylight.controller.cluster.datastore.Shard.finishCommit()
>> method has a comment that during edge cases data store will throw
>> IllegalStateException
>>
>> Is this behavior expected? What should we do when there is no owner for
>> the entity in the cluster?
>>
>>
>>
>> No Owner Event:
>>
>>
>>
>> 2017-01-27 22:13:25,920 | INFO | lt-dispatcher-16 |
>> EntityOwnerChangeListener | 283 - com.aaa.odl - 0.1.0.SNAPSHOT |
>> ownershipChanged:request,handleOwnershipChanged: EntityOwnershipChanged
>> [entity=Entity{type='controller', id=/(urn:opendaylight:params:x
>> ml:ns:yang:controller:md:sal:core:general-entity?revision=20
>> 15-08-20)entity/entity[{(urn:opendaylight:params:xml:ns:
>> yang:controller:md:sal:core:general-entity?revision=2015-08-20)name=controller}]},
>> wasOwner=false, isOwner=false, hasOwner=false, inJeopardy=false] event
>> received for entity Entity{type='controller', id=/(urn:opendaylight:params:x
>> ml:ns:yang:controller:md:sal:core:general-entity?revision=20
>> 15-08-20)entity/entity[{(urn:opendaylight:params:xml:ns:
>> yang:controller:md:sal:core:general-entity?revision=2015-
>> 08-20)name=controller}]}
>>
>>
>>
>>
>>
>> Exception:
>>
>>
>>
>> *2017-01-27 22:13:25,787 | WARN | lt-dispatcher-17 |
>> EntityOwnershipShard | 140 - org.opendaylight.controller.sa
>> <http://org.opendaylight.controller.sa>l-akka-raft - 1.3.3.Beryllium-SR3 |
>> member-2-shard-entity-ownership-operational: commit failed for transaction
>> member-2-txn-2-1485584005749 - retrying as foreign candidate*
>>
>> *java.util.concurrent.ExecutionException:
>> java.lang.IllegalStateException: Store tree
>> org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82
>> <org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82>
>> and candidate base
>> org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559
>> <org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559>
>> differ.*
>>
>> * at
>> com.google.common.util.concurrent.Futures$ImmediateFailedFuture.get(Futures.java:190)[37:com.google.guava:18.0.0]*
>>
>> * at org.opendaylight.controller.cl
>> <http://org.opendaylight.controller.cl>uster.datastore.ShardCommitCoordinator$CohortEntry.commit(ShardCommitCoordinator.java:670)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
>>
>> * at org.opendaylight.controller.cl
>> <http://org.opendaylight.controller.cl>uster.datastore.Shard.finishCommit(Shard.java:352)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
>>
>> * at org.opendaylight.controller.cl
>> <http://org.opendaylight.controller.cl>uster.datastore.Shard.finishCommit(Shard.java:420)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
>>
>> * at org.opendaylight.controller.cl
>> <http://org.opendaylight.controller.cl>uster.datastore.Shard.applyState(Shard.java:668)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
>>
>> * at org.opendaylight.controller.cl
>> <http://org.opendaylight.controller.cl>uster.raft.RaftActor.handleCommand(RaftActor.java:225)[140:org.opendaylight.controller.sal-akka-raft:1.3.3.Beryllium-SR3]*
>>
>> * at org.opendaylight.controller.cl
>> <http://org.opendaylight.controller.cl>uster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[139:org.opendaylight.controller.sal-clustering-commons:1.3.3.Beryllium-SR3]*
>>
>> * at org.opendaylight.controller.cl
>> <http://org.opendaylight.controller.cl>uster.datastore.Shard.onReceiveCommand(Shard.java:276)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
>>
>> * at org.opendaylight.controller.cl
>> <http://org.opendaylight.controller.cl>uster.datastore.entityownership.EntityOwnershipShard.onReceiveCommand(EntityOwnershipShard.java:137)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
>>
>> * at
>> akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
>>
>> * at org.opendaylight.controller.cl
>> <http://org.opendaylight.controller.cl>uster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[139:org.opendaylight.controller.sal-clustering-commons:1.3.3.Beryllium-SR3]*
>>
>> * at
>> akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[128:com.typesafe.akka.actor:2.3.14]*
>>
>> * at
>> akka.persistence.Recovery$State$class.process(Recovery.scala:30)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
>>
>> * at
>> akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
>>
>> * at
>> akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
>>
>> * at
>> akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
>>
>> * at
>> akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
>>
>> * at
>> akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
>>
>> * at
>> akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
>>
>> * at
>> akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
>>
>> * at
>> akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[133:com.typesafe.akka.persistence.experimental:2.3.14]*
>>
>> * at
>> akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[128:com.typesafe.akka.actor:2.3.14]*
>>
>> * at
>> akka.actor.ActorCell.invoke(ActorCell.scala:487)[128:com.typesafe.akka.actor:2.3.14]*
>>
>> * at
>> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[128:com.typesafe.akka.actor:2.3.14]*
>>
>> * at
>> akka.dispatch.Mailbox.run(Mailbox.scala:220)[128:com.typesafe.akka.actor:2.3.14]*
>>
>> * at
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[128:com.typesafe.akka.actor:2.3.14]*
>>
>> * at
>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
>>
>> * at
>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
>>
>> * at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
>>
>> * at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]*
>>
>> *Caused by: java.lang.IllegalStateException: Store tree
>> org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82
>> <org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@3316bc82>
>> and candidate base
>> org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559
>> <org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7a30a559>
>> differ.*
>>
>> * at
>> org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)[57:org.opendaylight.yangtools.yang-data-impl:0.8.3.Beryllium-SR3]*
>>
>> * at org.opendaylight.controller.cl
>> <http://org.opendaylight.controller.cl>uster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:106)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.3.Beryllium-SR3]*
>>
>> * ... 29 more*
>>
>>
>>
>> Thanks
>>
>>
>> _______________________________________________
>> controller-dev mailing list
>> controller-dev@lists.opendaylight.org
>> https://lists.opendaylight.org/mailman/listinfo/controller-dev
>>
>>
>>
>
>
--
Regards,
Siva.
_______________________________________________
controller-dev mailing list
controller-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/controller-dev
