Hi,
We have a project that is using Akka Persistence (2.3.4) with sharding in a
cluster (with 2 nodes).
It has 3 differnt types of AbstractPersistentActor all using Sharding.
When the cluster is running, everything works fine and it usually works to
stop the cluster and restart it again. But from time to time the following
happens when trying to start the cluster:
When starting the first node (server1), the logs gets flodded with
log-entries like this:
ERROR akka.actor.OneForOneStrategy
akka://NCSSystem/user/data/connectionCoordinator/singleton/coordinator
:
requirement failed: Region
Actor[akka.tcp://NCSSystem@server2/user/data/connection#789496176]
not registered: State(Map(),Map(),Set())
java.lang.IllegalArgumentException: requirement failed: Region Actor[
akka.tcp://NCSSystem@server2/user/data/connection#789496176] not
registered: State(Map(),Map(),Set())
at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.
4.jar:na]
at akka.contrib.pattern.ShardCoordinator$Internal$State.updated(
ClusterSharding.scala:1115) ~[akka-contrib_2.10-2.3.4.jar:2.3.4]
at akka.contrib.pattern.
ShardCoordinator$$anonfun$receiveRecover$1.applyOrElse(ClusterSharding.scala
:1236) ~[akka-contrib_2.10-2.3.4.jar:2.3.4]
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(
AbstractPartialFunction.scala:33) ~[scala-library-2.10.4.jar:na]
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(
AbstractPartialFunction.scala:33) ~[scala-library-2.10.4.jar:na]
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(
AbstractPartialFunction.scala:25) ~[scala-library-2.10.4.jar:na]
at akka.persistence.
Eventsourced$$anonfun$akka$persistence$Eventsourced$$recoveryBehavior$1.
applyOrElse(Eventsourced.scala:168) ~[akka-persistence-experimental_2.10-2.3
.4.jar:na]
at akka.persistence.Recovery$State$$anonfun$processPersistent$1.
apply(Recovery.scala:33) ~[akka-persistence-experimental_2.10-2.3.4.jar:na]
at akka.persistence.Recovery$State$$anonfun$processPersistent$1.
apply(Recovery.scala:33) ~[akka-persistence-experimental_2.10-2.3.4.jar:na]
at akka.persistence.Recovery$class.withCurrentPersistent(
Recovery.scala:176) ~[akka-persistence-experimental_2.10-2.3.4.jar:na]
at akka.contrib.pattern.ShardCoordinator.withCurrentPersistent(
ClusterSharding.scala:1192) ~[akka-contrib_2.10-2.3.4.jar:2.3.4]
at akka.persistence.Recovery$State$class.processPersistent(
Recovery.scala:33) ~[akka-persistence-experimental_2.10-2.3.4.jar:na]
at akka.persistence.Recovery$$anon$1.processPersistent(Recovery.
scala:95) ~[akka-persistence-experimental_2.10-2.3.4.jar:na]
at akka.persistence.Recovery$$anon$1.aroundReceive(Recovery.
scala:101) ~[akka-persistence-experimental_2.10-2.3.4.jar:na]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:
256) ~[akka-persistence-experimental_2.10-2.3.4.jar:na]
at akka.contrib.pattern.ShardCoordinator.
akka$persistence$Eventsourced$$super$aroundReceive(ClusterSharding.scala:
1192) ~[akka-contrib_2.10-2.3.4.jar:2.3.4]
at akka.persistence.Eventsourced$$anon$1.aroundReceive(
Eventsourced.scala:35) ~[akka-persistence-experimental_2.10-2.3.4.jar:na]
at akka.persistence.Eventsourced$class.aroundReceive(
Eventsourced.scala:369) ~[akka-persistence-experimental_2.10-2.3.4.jar:na]
at akka.contrib.pattern.ShardCoordinator.aroundReceive(
ClusterSharding.scala:1192) ~[akka-contrib_2.10-2.3.4.jar:2.3.4]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [
akka-actor_2.10-2.3.4.jar:na]
at akka.actor.ActorCell.invoke(ActorCell.scala:487) [akka-
actor_2.10-2.3.4.jar:na]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [akka
-actor_2.10-2.3.4.jar:na]
at akka.dispatch.Mailbox.run(Mailbox.scala:220) [akka-actor_2.10
-2.3.4.jar:na]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.
exec(AbstractDispatcher.scala:393) [akka-actor_2.10-2.3.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.
java:260) [scala-library-2.10.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(
ForkJoinPool.java:1339) [scala-library-2.10.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool
.java:1979) [scala-library-2.10.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
ForkJoinWorkerThread.java:107) [scala-library-2.10.4.jar:na]
This happens for all 3 coordinators and it does not look like it will never
stop logging. The connection#789496176 is not repeating.
The only way I have found to recover from this situation is to manually
delete all journal- and snapshot-entries for the coordinators.
Any help or hint would be appreciated.
Best regards,
Morten Kjetland
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
