Hi Patrik, I have managed to repro it twice again. We have typesafe support 
so I will get in touch with them to discuss how best to send the repro 
setup, as it's not a simple attachment!

thanks

Rich

On Monday, November 10, 2014 4:35:24 PM UTC, Patrik Nordwall wrote:
>
>
>
> On Mon, Nov 10, 2014 at 5:18 PM, Richard Bowker <
> [email protected] <javascript:>> wrote:
>
> Hi Patrik, unfortunately not.  In fact its only happened once to me so far 
> so may be a difficult one to reproduce.
>
> I will of course get back to you if I can find a trigger.
>
>
> Thanks. That is problematic with these bugs. I don't know much about 
> Cassandra. Is it possible to export the data from cassandra if this happens 
> again so it could be analyzed (replayed) by us? Given that you don't have 
> any sensitive information in it.
>
> /Patrik
>  
>
>
> Tackar!
>
> On Monday, November 10, 2014 4:04:18 PM UTC, Patrik Nordwall wrote:
>
> Hi Richard,
>
> That is not good. We have seen similar issue a few times and tracked it 
> down to bugs in the journal implementations. It will happen when events are 
> replayed in the wrong order.
>
> Is there a way we can reproduce this?
>
> Regards,
> Patrik
>
> On Mon, Nov 10, 2014 at 2:42 PM, Richard Bowker <mechajoh...@googlemail.
> com> wrote:
>
> I have had seen a similar problem when restarting nodes in a cluster using 
> sharding.
>
> after restarting, the node with the shard coordinator went into an 
> infinite error loop.
>
> I was using akka 2.3.6 and "com.github.krasserm" %% 
> "akka-persistence-cassandra" % "0.3.4" as the journal/persistence store.
>
> section of the error log below, I didn not know what to do to recover this 
> other than just manually delete all the akka keystores from the database 
> which obviously isn't ideal!
>
> any thoughts?
>
> thanks
>
> [ERROR] [11/10/2014 13:16:21.969] 
> [ClusterSystem-akka.actor.default-dispatcher-17] 
> [akka://ClusterSystem/user/sharding/PollServiceCoordinator/
> singleton/coordinator] 
>
> requirement failed: Region Actor[akka.tcp://ClusterSystem
> @172.31.18.169:2552/user/sharding/PollService#546005322] not registered: 
> State(Map(test47 -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test6 -> 
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test30 
> -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test42 -> 
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test29 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test18 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], 
>
> test14 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test36 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test25 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test28 -> 
>
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test43 
> -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], 
>
> test32 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test20 -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test15 -> 
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test33 -> 
>
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test22 
> -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test0 
>
> -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test44 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], 
>
> test11 -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test41 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test37 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test16 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test9 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test23 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test34 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test45 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test38 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test8 
>
> -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test19 -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test35 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test5 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test24 -> 
>
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test2 
> -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828] 
> <http://[email protected]:2552/user/sharding/PollService#1716980828%5D>
> ),Map
>
> (Actor[akka://ClusterSystem/user/sharding/PollService#1625036981] -> 
> Vector(test6, test18, test28, test20, test33, test44, test16, test34, 
> test8, test24, test5, test11, 
>
> test19, test25, test30, test38, test47), Actor[akka.tcp://ClusterSystem
> @172.31.21.9:2552/user/sharding/PollService#1716980828] -> Vector(test42, 
> test36, test32, test15, 
>
> test0, test41, test23, test45, test35, test2, test9, test14, test22, 
> test29, test37, test43)),Set(Actor
>
> [akka.tcp://[email protected]:49813/user/sharding/
> PollService#1785638107] 
> <http://[email protected]:49813/user/sharding/PollService#1785638107%5D>
> ))
> java.lang.IllegalArgumentException: requirement failed: Region 
> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#546005322] not registered: 
>
> State(Map(test47 -> 
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test6 -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test30 
>
> -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test42 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], 
>
> test29 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test18 -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test14 -> 
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test36 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test25 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], 
>
> test28 -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test43 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test32 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test20 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test15 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test33 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], 
>
> test22 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test0 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test44 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test11 -> 
>
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test41 
> -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], 
>
> test37 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test16 -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test9 -> 
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test23 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test34 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], 
>
> test45 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test38 -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test8 -> 
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test19 
> -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test35 -> 
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test5 -> 
>
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test24 
> -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test2 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828] 
> <http://[email protected]:2552/user/sharding/PollService#1716980828%5D>
> ),Map(Actor[akka://ClusterSystem/user/sharding/PollService#1625036981] -> 
> Vector(test6, 
>
> test18, test28, test20, test33, test44, test16, test34, test8, test24, 
> test5, test11, test19, test25, test30, test38, test47), Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828] -> Vector(test42, test36, test32, test15, test0, 
> test41, test23, test45, test35, test2, 
>
> test9, test14, test22, test29, test37, test43)),Set(Actor[akka.tcp://
> [email protected]:49813/user/sharding/PollService#1785638107] 
> <http://[email protected]:49813/user/sharding/PollService#1785638107%5D>
> ))
>         at scala.Predef$.require(Predef.scala:219)
>         at akka.contrib.pattern.ShardCoordinator$Internal$
> State.updated(ClusterSharding.scala:1115)
>         at akka.contrib.pattern.ShardCoordinator$$anonfun$
> receiveRecover$1.applyOrElse(ClusterSharding.scala:1236)
>         at scala.runtime.AbstractPartialFunction.apply(
> AbstractPartialFunction.scala:36)
>         at akka.persistence.Eventsourced$$anonfun$akka$persistence$
> Eventsourced$$recoveryBehavior$1.applyOrElse(Eventsourced.scala:168)
>         at akka.persistence.Recovery$class.runReceive(Recovery.scala:48)
>         at akka.contrib.pattern.ShardCoordinator.runReceive(
> ClusterSharding.scala:1192)
>         at akka.persistence.Recovery$State$$anonfun$
> processPersistent$1.apply(Recovery.scala:33)
>         at akka.persistence.Recovery$State$$anonfun$
> processPersistent$1.apply(Recovery.scala:33)
>         at akka.persistence.Recovery$class.withCurrentPersistent(
> Recovery.scala:185)
>         at akka.contrib.pattern.ShardCoordinator.withCurrentPersistent(
> ClusterSharding.scala:1192)
>         at akka.persistence.Recovery$State$class.processPersistent(
> Recovery.scala:33)
>         at akka.persistence.Recovery$$anon$1.processPersistent(
> Recovery.scala:104)
>         at akka.persistence.Recovery$$anon$1.aroundReceive(Recovery.
> scala:110)
>         at akka.persistence.Recovery$class.aroundReceive(Recovery.
> scala:265)
>         at akka.contrib.pattern.ShardCoordinator.akka$
> persistence$Eventsourced$$super$aroundReceive(ClusterSharding.scala:1192)
>         at akka.persistence.Eventsourced$$anon$1.aroundReceive(
> Eventsourced.scala:35)
>         at akka.persistence.Eventsourced$class.aroundReceive(
> Eventsourced.scala:369)
>         at akka.contrib.pattern.ShardCoordinator.
> aroundReceive(ClusterSharding.scala:1192)
>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>         at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>         at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>         at akka.dispatch.ForkJoinExecutorConfigurator$
> AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>         at scala.concurrent.forkjoin.ForkJoinTask.doExec(
> ForkJoinTask.java:260)
>         at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> runTask(ForkJoinPool.java:1339)
>         at scala.concurrent.forkjoin.ForkJoinPool.runWorker(
> ForkJoinPool.java:1979)
>         at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> ForkJoinWorkerThread.java:107)
>
> [ERROR] [11/10/2014 13:16:21.978] 
> [ClusterSystem-akka.actor.default-dispatcher-15] 
> [akka://ClusterSystem/user/sharding/PollServiceCoordinator/
> singleton/coordinator] 
>
> requirement failed: Region Actor[akka.tcp://ClusterSystem
> @172.31.18.169:2552/user/sharding/PollService#546005322] not registered: 
> State(Map(test47 -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test6 -> 
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test30 
> -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test42 -> 
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test29 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test18 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], 
>
> test14 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test36 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test25 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test28 -> 
>
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test43 
> -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], 
>
> test32 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test20 -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test15 -> 
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test33 -> 
>
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test22 
> -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test0 
>
> -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test44 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], 
>
> test11 -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test41 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test37 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test16 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test9 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test23 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test34 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test45 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test38 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test8 
>
> -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test19 -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test35 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test5 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test24 -> 
>
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test2 
> -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828] 
> <http://[email protected]:2552/user/sharding/PollService#1716980828%5D>
> ),Map
>
> (Actor[akka://ClusterSystem/user/sharding/PollService#1625036981] -> 
> Vector(test6, test18, test28, test20, test33, test44, test16, test34, 
> test8, test24, test5, test11, 
>
> test19, test25, test30, test38, test47), Actor[akka.tcp://ClusterSystem
> @172.31.21.9:2552/user/sharding/PollService#1716980828] -> Vector(test42, 
> test36, test32, test15, 
>
> test0, test41, test23, test45, test35, test2, test9, test14, test22, 
> test29, test37, test43)),Set(Actor
>
> [akka.tcp://[email protected]:49813/user/sharding/
> PollService#1785638107] 
> <http://[email protected]:49813/user/sharding/PollService#1785638107%5D>
> ))
> java.lang.IllegalArgumentException: requirement failed: Region 
> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#546005322] not registered: 
>
> State(Map(test47 -> 
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test6 -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test30 
>
> -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test42 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], 
>
> test29 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test18 -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test14 -> 
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test36 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test25 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], 
>
> test28 -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test43 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test32 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test20 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test15 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test33 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], 
>
> test22 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test0 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828], test44 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], test11 -> 
>
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test41 
> -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], 
>
> test37 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test16 -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test9 -> 
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test23 -> 
>
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test34 -> Actor[akka://ClusterSystem/
> user/sharding/PollService#1625036981], 
>
> test45 -> Actor[akka.tcp://[email protected]:2552/user/
> sharding/PollService#1716980828], test38 -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test8 -> 
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test19 
> -> Actor
>
> [akka://ClusterSystem/user/sharding/PollService#1625036981], test35 -> 
> Actor[akka.tcp://[email protected]:2552/user/sharding/PollService#
> 1716980828], test5 -> 
>
> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], test24 
> -> Actor[akka://ClusterSystem/user/sharding/PollService#1625036981], 
> test2 -> Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828] 
> <http://[email protected]:2552/user/sharding/PollService#1716980828%5D>
> ),Map(Actor[akka://ClusterSystem/user/sharding/PollService#1625036981] -> 
> Vector(test6, 
>
> test18, test28, test20, test33, test44, test16, test34, test8, test24, 
> test5, test11, test19, test25, test30, test38, test47), Actor
>
> [akka.tcp://[email protected]:2552/user/sharding/
> PollService#1716980828] -> Vector(test42, test36, test32, test15, test0, 
> test41, test23, test45, test35, test2, 
>
> test9, test14, test22, test29, test37, test43)),Set(Actor[akka.tcp://
> [email protected]:49813/user/sharding/PollService#1785638107] 
> <http://[email protected]:49813/user/sharding/PollService#1785638107%5D>
> ))
>         at scala.Predef$.require(Predef.scala:219)
>         at akka.contrib.pattern.ShardCoordinator$Internal$
> State.updated(ClusterSharding.scala:1115)
>         at akka.contrib.pattern.ShardCoordinator$$anonfun$
> receiveRecover$1.applyOrElse(ClusterSharding.scala:1236)
>         at scala.runtime.AbstractPartialFunction.apply(
> AbstractPartialFunction.scala:36)
>         at akka.persistence.Eventsourced$$anonfun$akka$persistence$
> Eventsourced$$recoveryBehavior$1.applyOrElse(Eventsourced.scala:168)
>         at akka.persistence.Recovery$class.runReceive(Recovery.scala:48)
>         at akka.contrib.pattern.ShardCoordinator.runReceive(
> ClusterSharding.scala:1192)
>         at akka.persistence.Recovery$State$$anonfun$
> processPersistent$1.apply(Recovery.scala:33)
>         at akka.persistence.Recovery$State$$anonfun$
> processPersistent$1.apply(Recovery.scala:33)
>         at akka.persistence.Recovery$class.withCurrentPersistent(
> Recovery.scala:185)
>         at akka.contrib.pattern.ShardCoordinator.withCurrentPersistent(
> ClusterSharding.scala:1192)
>         at akka.persistence.Recovery$State$class.processPersistent(
> Recovery.scala:33)
>         at akka.persistence.Recovery$$anon$1.processPersistent(
> Recovery.scala:104)
>         at akka.persistence.Recovery$$anon$1.aroundReceive(Recovery.
> scala:110)
>         at akka.persistence.Recovery$class.aroundReceive(Recovery.
> scala:265)
>         at akka.contrib.pattern.ShardCoordinator.akka$
> persistence$Eventsourced$$super$aroundReceive(ClusterSharding.scala:1192)
>         at akka.persistence.Eventsourced$$anon$1.aroundReceive(
> Eventsourced.scala:35)
>         at akka.persistence.Eventsourced$class.aroundReceive(
> Eventsourced.scala:369)
>         at akka.contrib.pattern.ShardCoordinator.
> aroundReceive(ClusterSharding.scala:1192)
>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>         at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>         at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>         at akka.dispatch.ForkJoinExecutorConfigurator$
> AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>         at scala.concurrent.forkjoin.ForkJoinTask.doExec(
> ForkJoinTask.java:260)
>         at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> runTask(ForkJoinPool.java:1339)
>         at scala.concurrent.forkjoin.ForkJoinPool.runWorker(
> ForkJoinPool.java:1979)
>         at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> ForkJoinWorkerThread.java:107)
>
>  
>
>  
>
> On Thursday, August 7, 2014 10:45:36 AM UTC+1, Konrad Malawski wrote:
>
> Great to hear you've found the problem!
> We'll provide a TCK for journal plugins with the next release (minor 
> already), so I suggest grinding your custom plugin with it to see if it's 
> really valid :-)
>
> Happy hakking!
>
>
> On Thu, Aug 7, 2014 at 11:30 AM, Morten Kjetland <[email protected]> 
> wrote:
>
> Hi,
>
> Turns out there was a bug in our homebrew jdbc-snapshot implementation.
>
> The loaded SelectedSnapshot was populated with Option(state) instead of 
> just the state, so the following lines in ShardCoordinator was not executed:
>
>  case SnapshotOffer(_, state: State) ⇒
>       log.debug("receiveRecover SnapshotOffer {}", state)
>       persistentState = state
>
> The snapshot was therefor never applied, so when it started receiving 
> events with sequenceNr after the snapshot, it blew up.
>
> Thanks a lot for helping me in the right direction.
>
> Best regards,
> Morten Kjetland
>
>
> On Wed, Aug 6, 2014 at 2:12 PM, Morten Kjetland <[email protected]> wrote:
>
> Thanks the response,
>
> We are using a homebrew jdbc journal.
>
> I checked the journal and ShardRegionProxyRegistered is written to it.
> But I was unable to reproduce the problem now.
> It might be a problem related to snapshoting in combination with a bug in 
> our jdbc journal.
> I'll try to reproduce it later and check the db again.
>
> I just saw that https://github.com/dnvriend/akka-persistence-jdbc was 
> worked on during the summer, so I'll try to use that one instead of our 
> own, and see if the problem goes away.
>
> Best regards,
> Morten Kjetland
>
>
> On Wed, Aug 6, 2014 at 12:40 PM, Konrad Malawski <[email protected]> 
> wrote:
>
> Hi Morten, 
> thanks for reporting!
> Which journal plugin are you using?
>
> It looks like during replay it gets an ShardHomeAllocated without getting 
> ShardRegionProxyRegistered first – which makes
>
> ...

-- 
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: 
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Reply via email to