Hi
We setup a cluster of 3 instances, and rebooted one of the instances.
During restart, it printed the following and went to a bad state where we
could not use the operational shard. Has anyone seen this akka exception
before? Is there any way to get out of this type of slate state?
2017-05-10 08:48:29,660 | WARN | saction-29-31'}} | DeadlockMonitor
| 125 - org.opendaylight.controller.config-manager -
0.5.2.Boron-SR2 | ModuleIdentifier{factoryName='runtime-generated-mapping',
instanceName='runtime-mapping-singleton'} did not finish after 18998 ms
2017-05-10 08:48:32,089 | ERROR | ult-dispatcher-4 | Shard
| 188 - org.opendaylight.controller.sal-akka-raft -
1.4.2.Boron-SR2 | member-1-shard-default-operational: failed to apply
payload
org.opendaylight.controller.cluster.datastore.persisted.CommitTransactionPayload@e4be907
java.lang.IllegalStateException: member-1-shard-default-operational: Failed
to apply recovery payload. Modification data was written to file
/opt/Xceed/Xceed_controller/data/failed-recovery-payload-member-1-shard-default-operational.out
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryCandidate(ShardDataTree.java:267)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryPayload(ShardDataTree.java:284)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.datastore.ShardRecoveryCoordinator.appendRecoveredLogEntry(ShardRecoveryCoordinator.java:56)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.batchRecoveredLogEntry(RaftActorRecoverySupport.java:233)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.onRecoveredApplyLogEntries(RaftActorRecoverySupport.java:204)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.handleRecoveryMessage(RaftActorRecoverySupport.java:66)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.RaftActor.handleRecover(RaftActor.java:167)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.datastore.Shard.handleRecover(Shard.java:221)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveRecover(AbstractUntypedPersistentActor.java:38)[187:org.opendaylight.controller.sal-clustering-commons:1.4.2.Boron-SR2]
at
akka.persistence.UntypedPersistentActor$$anonfun$receiveRecover$1.applyOrElse(PersistentActor.scala:173)[181:com.typesafe.akka.persistence:2.4.7]
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
akka.persistence.Eventsourced$$anon$3$$anonfun$1.applyOrElse(Eventsourced.scala:475)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.actor.Actor$class.aroundReceive(Actor.scala:484)[175:com.typesafe.akka.actor:2.4.7]
at
akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$$anon$4.stateReceive(Eventsourced.scala:527)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:179)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[175:com.typesafe.akka.actor:2.4.7]
at
akka.actor.ActorCell.invoke(ActorCell.scala:495)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.run(Mailbox.scala:224)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.exec(Mailbox.scala:234)[175:com.typesafe.akka.actor:2.4.7]
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
Caused by:
org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException:
Node
/(urn:ietf:params:xml:ns:yang:ietf-network?revision=2015-12-08)networks/network/network[{(urn:ietf:params:xml:ns:yang:ietf-network?revision=2015-12-08)network-id=urn:ogf:network:network=network1}]/node[{(urn:ietf:params:xml:ns:yang:ietf-network?revision=2015-12-08)node-id=urn:ogf:network:network=network1:node=nodeA}]
does not exist. Cannot apply modification to its children.
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:281)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryCandidate(ShardDataTree.java:261)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
... 25 more
2017-05-10 08:48:32,098 | ERROR | lt-dispatcher-22 | Shard
| 176 - com.typesafe.akka.slf4j - 2.4.7 | Exception in
receiveRecover when replaying event type
[org.opendaylight.controller.cluster.raft.persisted.ApplyJournalEntries]
with sequence number [3724] for persistenceId
[member-1-shard-default-operational].
java.lang.IllegalStateException: member-1-shard-default-operational: Failed
to apply recovery payload
org.opendaylight.controller.cluster.datastore.persisted.CommitTransactionPayload@e4be907
at
org.opendaylight.controller.cluster.datastore.ShardRecoveryCoordinator.appendRecoveredLogEntry(ShardRecoveryCoordinator.java:59)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.batchRecoveredLogEntry(RaftActorRecoverySupport.java:233)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.onRecoveredApplyLogEntries(RaftActorRecoverySupport.java:204)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.handleRecoveryMessage(RaftActorRecoverySupport.java:66)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.RaftActor.handleRecover(RaftActor.java:167)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.datastore.Shard.handleRecover(Shard.java:221)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveRecover(AbstractUntypedPersistentActor.java:38)[187:org.opendaylight.controller.sal-clustering-commons:1.4.2.Boron-SR2]
at
akka.persistence.UntypedPersistentActor$$anonfun$receiveRecover$1.applyOrElse(PersistentActor.scala:173)[181:com.typesafe.akka.persistence:2.4.7]
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
akka.persistence.Eventsourced$$anon$3$$anonfun$1.applyOrElse(Eventsourced.scala:475)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.actor.Actor$class.aroundReceive(Actor.scala:484)[175:com.typesafe.akka.actor:2.4.7]
at
akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$$anon$4.stateReceive(Eventsourced.scala:527)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:179)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[175:com.typesafe.akka.actor:2.4.7]
at
akka.actor.ActorCell.invoke(ActorCell.scala:495)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.run(Mailbox.scala:224)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.exec(Mailbox.scala:234)[175:com.typesafe.akka.actor:2.4.7]
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
Caused by: java.lang.IllegalStateException:
member-1-shard-default-operational: Failed to apply recovery payload.
Modification data was written to file
/opt/Xceed/Xceed_controller/data/failed-recovery-payload-member-1-shard-default-operational.out
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryCandidate(ShardDataTree.java:267)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryPayload(ShardDataTree.java:284)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.datastore.ShardRecoveryCoordinator.appendRecoveredLogEntry(ShardRecoveryCoordinator.java:56)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
... 23 more
Caused by:
org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException:
Node
/(urn:ietf:params:xml:ns:yang:ietf-network?revision=2015-12-08)networks/network/network[{(urn:ietf:params:xml:ns:yang:ietf-network?revision=2015-12-08)network-id=urn:ogf:network:network=network1}]/node[{(urn:ietf:params:xml:ns:yang:ietf-network?revision=2015-12-08)node-id=urn:ogf:network:network=network1:node=nodeA}]
does not exist. Cannot apply modification to its children.
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:281)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35)[79:org.opendaylight.yangtools.yang-data-impl:1.0.2.Boron-SR2]
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryCandidate(ShardDataTree.java:261)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
... 25 more
2017-05-10 08:48:32,104 | INFO | lt-dispatcher-20 | Shard
| 188 - org.opendaylight.controller.sal-akka-raft -
1.4.2.Boron-SR2 | Stopping Shard member-1-shard-default-operational
Thanks
Srini.
_______________________________________________
controller-dev mailing list
controller-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/controller-dev
