Hi Controller-dev,
We have a three-node cluster running from some time. When one of the nodes is
restarted today, we see below exceptions and the shard
"member-3-shard-ensemble-operational" has not come up at all.
Below operations from the cluster-admin yang have been tried, but did not help
and they are failed with AskTimeoutExceptions
/operations/cluster-admin:remove-shard-replica
2018-10-23 11:46:58,895 | ERROR | lt-dispatcher-34 | ClusterAdminRpcService
| 171 - org.opendaylight.controller.sal-cluster-admin - 1.4.3.Boron-SR3 |
Failed to remove replica for shard ensemble
akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka://opendaylight-cluster-data/user/shardmanager-operational#1512050007]]
after [60000 ms]. Sender[null] sent message of type
"org.opendaylight.controller.cluster.datastore.messages.RemoveShardReplica".
at
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)[152:com.typesafe.akka.actor:2.4.7]
at
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)[152:com.typesafe.akka.actor:2.4.7]
at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:331)[152:com.typesafe.akka.actor:2.4.7]
at
akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:282)[152:com.typesafe.akka.actor:2.4.7]
at
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:286)[152:com.typesafe.akka.actor:2.4.7]
at
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)[152:com.typesafe.akka.actor:2.4.7]
/operations/cluster-admin:add-shard-replica
2018-10-23 11:49:23,694 | ERROR | ult-dispatcher-5 | ClusterAdminRpcService
| 171 - org.opendaylight.controller.sal-cluster-admin - 1.4.3.Boron-SR3 |
Failed to add replica for shard ensemble
akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka://opendaylight-cluster-data/user/shardmanager-operational#1512050007]]
after [60000 ms]. Sender[null] sent message of type
"org.opendaylight.controller.cluster.datastore.messages.AddShardReplica".
at
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)[152:com.typesafe.akka.actor:2.4.7]
at
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)[152:com.typesafe.akka.actor:2.4.7]
at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:331)[152:com.typesafe.akka.actor:2.4.7]
at
akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:282)[152:com.typesafe.akka.actor:2.4.7]
at
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:286)[152:com.typesafe.akka.actor:2.4.7]
at
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)[152:com.typesafe.akka.actor:2.4.7]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]
Modification data was written to file
/opt/controller-18.1.1.9999_ESMBD-7544_secure_cloud_2.128/data/failed-recovery-payload-member-3-shard-ensemble-operational.out
Can this file be used in any way to recover from the state?
Please let us know how can we recover the shard from this state?
2018-10-23 07:24:04,454 | ERROR | ult-dispatcher-4 | Shard
| 165 - org.opendaylight.controller.sal-akka-raft - 1.4.3.Boron-SR3 |
member-3-shard-ensemble-operational: failed to apply payload
org.opendaylight.controller.cluster.datastore.persisted.CommitTransactionPayload@6eceae70
java.lang.IllegalStateException: member-3-shard-ensemble-operational: Failed to
apply recovery payload. Modification data was written to file
/opt/controller-18.1.1.9999_ESMBD-7544_secure_cloud_2.128/data/failed-recovery-payload-member-3-shard-ensemble-operational.out
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryCandidate(ShardDataTree.java:267)[170:org.opendaylight.controller.sal-distributed-datastore:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryPayload(ShardDataTree.java:284)[170:org.opendaylight.controller.sal-distributed-datastore:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.datastore.ShardRecoveryCoordinator.appendRecoveredLogEntry(ShardRecoveryCoordinator.java:56)[170:org.opendaylight.controller.sal-distributed-datastore:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.batchRecoveredLogEntry(RaftActorRecoverySupport.java:233)[165:org.opendaylight.controller.sal-akka-raft:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.onRecoveredApplyLogEntries(RaftActorRecoverySupport.java:204)[165:org.opendaylight.controller.sal-akka-raft:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.handleRecoveryMessage(RaftActorRecoverySupport.java:66)[165:org.opendaylight.controller.sal-akka-raft:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.raft.RaftActor.handleRecover(RaftActor.java:167)[165:org.opendaylight.controller.sal-akka-raft:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.datastore.Shard.handleRecover(Shard.java:221)[170:org.opendaylight.controller.sal-distributed-datastore:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveRecover(AbstractUntypedPersistentActor.java:38)[164:org.opendaylight.controller.sal-clustering-commons:1.4.3.Boron-SR3]
at
akka.persistence.UntypedPersistentActor$$anonfun$receiveRecover$1.applyOrElse(PersistentActor.scala:173)[158:com.typesafe.akka.persistence:2.4.7]
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
akka.persistence.Eventsourced$$anon$3$$anonfun$1.applyOrElse(Eventsourced.scala:475)[158:com.typesafe.akka.persistence:2.4.7]
at
akka.actor.Actor$class.aroundReceive(Actor.scala:484)[152:com.typesafe.akka.actor:2.4.7]
at
akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[158:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$$anon$4.stateReceive(Eventsourced.scala:527)[158:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:179)[158:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[158:com.typesafe.akka.persistence:2.4.7]
at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[152:com.typesafe.akka.actor:2.4.7]
at
akka.actor.ActorCell.invoke(ActorCell.scala:495)[152:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[152:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.run(Mailbox.scala:224)[152:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.exec(Mailbox.scala:234)[152:com.typesafe.akka.actor:2.4.7]
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
Caused by:
org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException:
Node
/(http://www.advaoptical.com/ensemble/controller/controllerconfiguration/core?revision=2016-09-22)ensemble-controller/AugmentationIdentifier{childNames=[(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)network-applications,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)connector-network-groups,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)interfaces,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)network-groups,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)networks,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)network-groups-deletion-status,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)connector-interfaces,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)network-groups-deletion-flows-info,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)lan-service-ports,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)network-tunnels,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)links]}/(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)connector-interfaces/connector-interface/connector-interface[{(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)connector-uid=CNODE834}]
does not exist. Cannot apply modification to its children.
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:281)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryCandidate(ShardDataTree.java:261)[170:org.opendaylight.controller.sal-distributed-datastore:1.4.3.Boron-SR3]
... 25 more
2018-10-23 07:24:04,468 | ERROR | lt-dispatcher-20 | Shard
| 153 - com.typesafe.akka.slf4j - 2.4.7 | Exception in receiveRecover
when replaying event type
[org.opendaylight.controller.cluster.raft.persisted.ApplyJournalEntries] with
sequence number [39972] for persistenceId [member-3-shard-ensemble-operational].
java.lang.IllegalStateException: member-3-shard-ensemble-operational: Failed to
apply recovery payload
org.opendaylight.controller.cluster.datastore.persisted.CommitTransactionPayload@6eceae70
at
org.opendaylight.controller.cluster.datastore.ShardRecoveryCoordinator.appendRecoveredLogEntry(ShardRecoveryCoordinator.java:59)[170:org.opendaylight.controller.sal-distributed-datastore:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.batchRecoveredLogEntry(RaftActorRecoverySupport.java:233)[165:org.opendaylight.controller.sal-akka-raft:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.onRecoveredApplyLogEntries(RaftActorRecoverySupport.java:204)[165:org.opendaylight.controller.sal-akka-raft:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.raft.RaftActorRecoverySupport.handleRecoveryMessage(RaftActorRecoverySupport.java:66)[165:org.opendaylight.controller.sal-akka-raft:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.raft.RaftActor.handleRecover(RaftActor.java:167)[165:org.opendaylight.controller.sal-akka-raft:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.datastore.Shard.handleRecover(Shard.java:221)[170:org.opendaylight.controller.sal-distributed-datastore:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveRecover(AbstractUntypedPersistentActor.java:38)[164:org.opendaylight.controller.sal-clustering-commons:1.4.3.Boron-SR3]
at
akka.persistence.UntypedPersistentActor$$anonfun$receiveRecover$1.applyOrElse(PersistentActor.scala:173)[158:com.typesafe.akka.persistence:2.4.7]
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
akka.persistence.Eventsourced$$anon$3$$anonfun$1.applyOrElse(Eventsourced.scala:475)[158:com.typesafe.akka.persistence:2.4.7]
at
akka.actor.Actor$class.aroundReceive(Actor.scala:484)[152:com.typesafe.akka.actor:2.4.7]
at
akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[158:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$$anon$4.stateReceive(Eventsourced.scala:527)[158:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:179)[158:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[158:com.typesafe.akka.persistence:2.4.7]
at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[152:com.typesafe.akka.actor:2.4.7]
at
akka.actor.ActorCell.invoke(ActorCell.scala:495)[152:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[152:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.run(Mailbox.scala:224)[152:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.exec(Mailbox.scala:234)[152:com.typesafe.akka.actor:2.4.7]
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[148:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
Caused by: java.lang.IllegalStateException:
member-3-shard-ensemble-operational: Failed to apply recovery payload.
Modification data was written to file
/opt/controller-18.1.1.9999_ESMBD-7544_secure_cloud_2.128/data/failed-recovery-payload-member-3-shard-ensemble-operational.out
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryCandidate(ShardDataTree.java:267)[170:org.opendaylight.controller.sal-distributed-datastore:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryPayload(ShardDataTree.java:284)[170:org.opendaylight.controller.sal-distributed-datastore:1.4.3.Boron-SR3]
at
org.opendaylight.controller.cluster.datastore.ShardRecoveryCoordinator.appendRecoveredLogEntry(ShardRecoveryCoordinator.java:56)[170:org.opendaylight.controller.sal-distributed-datastore:1.4.3.Boron-SR3]
... 23 more
Caused by:
org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException:
Node
/(http://www.advaoptical.com/ensemble/controller/controllerconfiguration/core?revision=2016-09-22)ensemble-controller/AugmentationIdentifier{childNames=[(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)network-applications,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)connector-network-groups,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)interfaces,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)network-groups,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)networks,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)network-groups-deletion-status,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)connector-interfaces,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)network-groups-deletion-flows-info,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)lan-service-ports,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)network-tunnels,
(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)links]}/(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)connector-interfaces/connector-interface/connector-interface[{(http://www.advaoptical.com/ensemble/controller/controller-network-configs?revision=2017-08-01)connector-uid=CNODE834}]
does not exist. Cannot apply modification to its children.
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:281)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.Boron-SR3]
at
org.opendaylight.controller.cluster.datastore.ShardDataTree.applyRecoveryCandidate(ShardDataTree.java:261)[170:org.opendaylight.controller.sal-distributed-datastore:1.4.3.Boron-SR3]
... 25 more
2018-10-23 07:24:04,476 | INFO | ult-dispatcher-3 | Shard
| 165 - org.opendaylight.controller.sal-akka-raft - 1.4.3.Boron-SR3 |
Stopping Shard member-3-shard-ensemble-operational
Thanks
Vikram
_______________________________________________
controller-dev mailing list
controller-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/controller-dev
