Srini, Looks like you have operational persistence on. It seems something must have changed between the restart. Upgrade? Data model change?
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree. ModifiedNodeDoesNotExistException: Node /(urn:ietf:params:xml:ns:yang: ietf-network?revision=2015-12-08)networks/network/network[{( urn:ietf:params:xml:ns:yang:ietf-network?revision=2015-12- 08)network-id=urn:ogf:network:network=network1}]/node[{(urn: ietf:params:xml:ns:yang:ietf-network?revision=2015-12-08) node-id=urn:ogf:network:network=network1:node=nodeA}] does not exist. Cannot apply modification to its children. It prints the offending data to am XML file (/opt/Xceed/Xceed_controller/ data/failed-recovery-payload-member-1-shard-default-operational.out) so one can view the offending data. Tom On Mon, May 15, 2017 at 12:53 PM, Srini Seetharaman < srini.seethara...@gmail.com> wrote: > Hi > We setup a cluster of 3 instances, and rebooted one of the instances. > During restart, it printed the following and went to a bad state where we > could not use the operational shard. Has anyone seen this akka exception > before? Is there any way to get out of this type of slate state? > > > 2017-05-10 08:48:29,660 | WARN | saction-29-31'}} | DeadlockMonitor > | 125 - org.opendaylight.controller.config-manager - > 0.5.2.Boron-SR2 | ModuleIdentifier{factoryName='runtime-generated-mapping', > instanceName='runtime-mapping-singleton'} did not finish after 18998 ms > 2017-05-10 08:48:32,089 | ERROR | ult-dispatcher-4 | Shard > | 188 - org.opendaylight.controller.sal-akka-raft - > 1.4.2.Boron-SR2 | member-1-shard-default-operational: failed to apply > payload org.opendaylight.controller.cluster.datastore.persisted. > CommitTransactionPayload@e4be907 > java.lang.IllegalStateException: member-1-shard-default-operational: > Failed to apply recovery payload. Modification data was written to file > /opt/Xceed/Xceed_controller/data/failed-recovery-payload- > member-1-shard-default-operational.out > at org.opendaylight.controller.cluster.datastore.ShardDataTree. > applyRecoveryCandidate(ShardDataTree.java:267)[193: > org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.datastore.ShardDataTree. > applyRecoveryPayload(ShardDataTree.java:284)[193: > org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.datastore. > ShardRecoveryCoordinator.appendRecoveredLogEntry( > ShardRecoveryCoordinator.java:56)[193:org.opendaylight. > controller.sal-distributed-datastore:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.raft. > RaftActorRecoverySupport.batchRecoveredLogEntry( > RaftActorRecoverySupport.java:233)[188:org.opendaylight. > controller.sal-akka-raft:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.raft. > RaftActorRecoverySupport.onRecoveredApplyLogEntries( > RaftActorRecoverySupport.java:204)[188:org.opendaylight. > controller.sal-akka-raft:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.raft. > RaftActorRecoverySupport.handleRecoveryMessage( > RaftActorRecoverySupport.java:66)[188:org.opendaylight. > controller.sal-akka-raft:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.raft.RaftActor. > handleRecover(RaftActor.java:167)[188:org.opendaylight. > controller.sal-akka-raft:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.datastore.Shard. > handleRecover(Shard.java:221)[193:org.opendaylight. > controller.sal-distributed-datastore:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.common.actor. > AbstractUntypedPersistentActor.onReceiveRecover( > AbstractUntypedPersistentActor.java:38)[187:org. > opendaylight.controller.sal-clustering-commons:1.4.2.Boron-SR2] > at akka.persistence.UntypedPersistentActor$$ > anonfun$receiveRecover$1.applyOrElse(PersistentActor. > scala:173)[181:com.typesafe.akka.persistence:2.4.7] > at scala.runtime.AbstractPartialFunction.apply( > AbstractPartialFunction.scala:36)[171:org.scala-lang.scala- > library:2.11.8.v20160304-115712-1706a37eb8] > at akka.persistence.Eventsourced$$anon$3$$anonfun$1. > applyOrElse(Eventsourced.scala:475)[181:com.typesafe. > akka.persistence:2.4.7] > at akka.actor.Actor$class.aroundReceive(Actor.scala:484) > [175:com.typesafe.akka.actor:2.4.7] > at akka.persistence.UntypedPersistentActor.akka$ > persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[ > 181:com.typesafe.akka.persistence:2.4.7] > at akka.persistence.Eventsourced$$anon$4.stateReceive( > Eventsourced.scala:527)[181:com.typesafe.akka.persistence:2.4.7] > at akka.persistence.Eventsourced$class.aroundReceive( > Eventsourced.scala:179)[181:com.typesafe.akka.persistence:2.4.7] > at akka.persistence.UntypedPersistentActor. > aroundReceive(PersistentActor.scala:168)[181:com.typesafe. > akka.persistence:2.4.7] > at akka.actor.ActorCell.receiveMessage(ActorCell. > scala:526)[175:com.typesafe.akka.actor:2.4.7] > at akka.actor.ActorCell.invoke(ActorCell.scala:495)[175:com. > typesafe.akka.actor:2.4.7] > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala: > 257)[175:com.typesafe.akka.actor:2.4.7] > at akka.dispatch.Mailbox.run(Mailbox.scala:224)[175:com. > typesafe.akka.actor:2.4.7] > at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[175:com. > typesafe.akka.actor:2.4.7] > at scala.concurrent.forkjoin.ForkJoinTask.doExec( > ForkJoinTask.java:260)[171:org.scala-lang.scala-library: > 2.11.8.v20160304-115712-1706a37eb8] > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue. > runTask(ForkJoinPool.java:1339)[171:org.scala-lang.scala-library:2.11.8. > v20160304-115712-1706a37eb8] > at scala.concurrent.forkjoin.ForkJoinPool.runWorker( > ForkJoinPool.java:1979)[171:org.scala-lang.scala-library: > 2.11.8.v20160304-115712-1706a37eb8] > at scala.concurrent.forkjoin.ForkJoinWorkerThread.run( > ForkJoinWorkerThread.java:107)[171:org.scala-lang.scala- > library:2.11.8.v20160304-115712-1706a37eb8] > Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree. > ModifiedNodeDoesNotExistException: Node /(urn:ietf:params:xml:ns:yang: > ietf-network?revision=2015-12-08)networks/network/network[{( > urn:ietf:params:xml:ns:yang:ietf-network?revision=2015-12- > 08)network-id=urn:ogf:network:network=network1}]/node[{(urn: > ietf:params:xml:ns:yang:ietf-network?revision=2015-12-08) > node-id=urn:ogf:network:network=network1:node=nodeA}] does not exist. > Cannot apply modification to its children. > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkTouchApplicable( > AbstractNodeContainerModificationStrategy.java:281)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation. > java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkChildPreconditions( > AbstractNodeContainerModificationStrategy.java:305)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkTouchApplicable( > AbstractNodeContainerModificationStrategy.java:288)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation. > java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkChildPreconditions( > AbstractNodeContainerModificationStrategy.java:305)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkTouchApplicable( > AbstractNodeContainerModificationStrategy.java:288)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation. > java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkChildPreconditions( > AbstractNodeContainerModificationStrategy.java:305)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkTouchApplicable( > AbstractNodeContainerModificationStrategy.java:288)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation. > java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > StructuralContainerModificationStrategy.checkApplicable( > StructuralContainerModificationStrategy.java:99)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkChildPreconditions( > AbstractNodeContainerModificationStrategy.java:305)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkTouchApplicable( > AbstractNodeContainerModificationStrategy.java:288)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation. > java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > RootModificationApplyOperation.checkApplicable( > RootModificationApplyOperation.java:72)[79:org.opendaylight. > yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35)[ > 79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.controller.cluster.datastore.ShardDataTree. > applyRecoveryCandidate(ShardDataTree.java:261)[193: > org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2] > ... 25 more > 2017-05-10 08:48:32,098 | ERROR | lt-dispatcher-22 | Shard > | 176 - com.typesafe.akka.slf4j - 2.4.7 | Exception in > receiveRecover when replaying event type [org.opendaylight.controller. > cluster.raft.persisted.ApplyJournalEntries] with sequence number [3724] > for persistenceId [member-1-shard-default-operational]. > java.lang.IllegalStateException: member-1-shard-default-operational: > Failed to apply recovery payload org.opendaylight.controller. > cluster.datastore.persisted.CommitTransactionPayload@e4be907 > at org.opendaylight.controller.cluster.datastore. > ShardRecoveryCoordinator.appendRecoveredLogEntry( > ShardRecoveryCoordinator.java:59)[193:org.opendaylight. > controller.sal-distributed-datastore:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.raft. > RaftActorRecoverySupport.batchRecoveredLogEntry( > RaftActorRecoverySupport.java:233)[188:org.opendaylight. > controller.sal-akka-raft:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.raft. > RaftActorRecoverySupport.onRecoveredApplyLogEntries( > RaftActorRecoverySupport.java:204)[188:org.opendaylight. > controller.sal-akka-raft:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.raft. > RaftActorRecoverySupport.handleRecoveryMessage( > RaftActorRecoverySupport.java:66)[188:org.opendaylight. > controller.sal-akka-raft:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.raft.RaftActor. > handleRecover(RaftActor.java:167)[188:org.opendaylight. > controller.sal-akka-raft:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.datastore.Shard. > handleRecover(Shard.java:221)[193:org.opendaylight. > controller.sal-distributed-datastore:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.common.actor. > AbstractUntypedPersistentActor.onReceiveRecover( > AbstractUntypedPersistentActor.java:38)[187:org. > opendaylight.controller.sal-clustering-commons:1.4.2.Boron-SR2] > at akka.persistence.UntypedPersistentActor$$ > anonfun$receiveRecover$1.applyOrElse(PersistentActor. > scala:173)[181:com.typesafe.akka.persistence:2.4.7] > at scala.runtime.AbstractPartialFunction.apply( > AbstractPartialFunction.scala:36)[171:org.scala-lang.scala- > library:2.11.8.v20160304-115712-1706a37eb8] > at akka.persistence.Eventsourced$$anon$3$$anonfun$1. > applyOrElse(Eventsourced.scala:475)[181:com.typesafe. > akka.persistence:2.4.7] > at akka.actor.Actor$class.aroundReceive(Actor.scala:484) > [175:com.typesafe.akka.actor:2.4.7] > at akka.persistence.UntypedPersistentActor.akka$ > persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[ > 181:com.typesafe.akka.persistence:2.4.7] > at akka.persistence.Eventsourced$$anon$4.stateReceive( > Eventsourced.scala:527)[181:com.typesafe.akka.persistence:2.4.7] > at akka.persistence.Eventsourced$class.aroundReceive( > Eventsourced.scala:179)[181:com.typesafe.akka.persistence:2.4.7] > at akka.persistence.UntypedPersistentActor. > aroundReceive(PersistentActor.scala:168)[181:com.typesafe. > akka.persistence:2.4.7] > at akka.actor.ActorCell.receiveMessage(ActorCell. > scala:526)[175:com.typesafe.akka.actor:2.4.7] > at akka.actor.ActorCell.invoke(ActorCell.scala:495)[175:com. > typesafe.akka.actor:2.4.7] > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala: > 257)[175:com.typesafe.akka.actor:2.4.7] > at akka.dispatch.Mailbox.run(Mailbox.scala:224)[175:com. > typesafe.akka.actor:2.4.7] > at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[175:com. > typesafe.akka.actor:2.4.7] > at scala.concurrent.forkjoin.ForkJoinTask.doExec( > ForkJoinTask.java:260)[171:org.scala-lang.scala-library: > 2.11.8.v20160304-115712-1706a37eb8] > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue. > runTask(ForkJoinPool.java:1339)[171:org.scala-lang.scala-library:2.11.8. > v20160304-115712-1706a37eb8] > at scala.concurrent.forkjoin.ForkJoinPool.runWorker( > ForkJoinPool.java:1979)[171:org.scala-lang.scala-library: > 2.11.8.v20160304-115712-1706a37eb8] > at scala.concurrent.forkjoin.ForkJoinWorkerThread.run( > ForkJoinWorkerThread.java:107)[171:org.scala-lang.scala- > library:2.11.8.v20160304-115712-1706a37eb8] > Caused by: java.lang.IllegalStateException: > member-1-shard-default-operational: > Failed to apply recovery payload. Modification data was written to file > /opt/Xceed/Xceed_controller/data/failed-recovery-payload- > member-1-shard-default-operational.out > at org.opendaylight.controller.cluster.datastore.ShardDataTree. > applyRecoveryCandidate(ShardDataTree.java:267)[193: > org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.datastore.ShardDataTree. > applyRecoveryPayload(ShardDataTree.java:284)[193: > org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2] > at org.opendaylight.controller.cluster.datastore. > ShardRecoveryCoordinator.appendRecoveredLogEntry( > ShardRecoveryCoordinator.java:56)[193:org.opendaylight. > controller.sal-distributed-datastore:1.4.2.Boron-SR2] > ... 23 more > Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree. > ModifiedNodeDoesNotExistException: Node /(urn:ietf:params:xml:ns:yang: > ietf-network?revision=2015-12-08)networks/network/network[{( > urn:ietf:params:xml:ns:yang:ietf-network?revision=2015-12- > 08)network-id=urn:ogf:network:network=network1}]/node[{(urn: > ietf:params:xml:ns:yang:ietf-network?revision=2015-12-08) > node-id=urn:ogf:network:network=network1:node=nodeA}] does not exist. > Cannot apply modification to its children. > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkTouchApplicable( > AbstractNodeContainerModificationStrategy.java:281)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation. > java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkChildPreconditions( > AbstractNodeContainerModificationStrategy.java:305)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkTouchApplicable( > AbstractNodeContainerModificationStrategy.java:288)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation. > java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkChildPreconditions( > AbstractNodeContainerModificationStrategy.java:305)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkTouchApplicable( > AbstractNodeContainerModificationStrategy.java:288)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation. > java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkChildPreconditions( > AbstractNodeContainerModificationStrategy.java:305)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkTouchApplicable( > AbstractNodeContainerModificationStrategy.java:288)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation. > java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > StructuralContainerModificationStrategy.checkApplicable( > StructuralContainerModificationStrategy.java:99)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkChildPreconditions( > AbstractNodeContainerModificationStrategy.java:305)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractNodeContainerModificationStrategy.checkTouchApplicable( > AbstractNodeContainerModificationStrategy.java:288)[79:org. > opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation. > java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > RootModificationApplyOperation.checkApplicable( > RootModificationApplyOperation.java:72)[79:org.opendaylight. > yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.yangtools.yang.data.impl.schema.tree. > AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35)[ > 79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2] > at org.opendaylight.controller.cluster.datastore.ShardDataTree. > applyRecoveryCandidate(ShardDataTree.java:261)[193: > org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2] > ... 25 more > 2017-05-10 08:48:32,104 | INFO | lt-dispatcher-20 | Shard > | 188 - org.opendaylight.controller.sal-akka-raft - > 1.4.2.Boron-SR2 | Stopping Shard member-1-shard-default-operational > > > Thanks > Srini. > > _______________________________________________ > controller-dev mailing list > controller-dev@lists.opendaylight.org > https://lists.opendaylight.org/mailman/listinfo/controller-dev > >
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev