Thanks for the update. For reference: Removal of Internal Cluster Sharding Data <http://doc.akka.io/docs/akka/2.4.0/scala/cluster-sharding.html#Removal_of_Internal_Cluster_Sharding_Data>
/Patrik On Thu, Oct 29, 2015 at 11:49 PM, Stefan Bleibinhaus < [email protected]> wrote: > Update: I was able to recover after deleting some messages. I also found > this relevant issue https://github.com/akka/akka/issues/17955 > > I deleted the events persisted by the second cluster and all persistent > actors affected by it (that's probably optional and depends on your data > model), but what really solved it was deleting the coordinator messages. > The *persistent_id* s of the coordinators look like this: > */user/sharding/YourPersistentActorCoordinator/singleton/coordinator* , > */user/sharding/YourActorViewCoordinator/singleton/coordinator* and are > stored alongside your regular persisted events in the *messages* table. > > > On Thursday, October 29, 2015 at 2:10:47 PM UTC-7, Stefan Bleibinhaus > wrote: >> >> I ran into the same issue in a staging environment with Akka 2.3.12 and >> Cassandra as persistent backend. I did what Patrik suggested and believe it >> is triggered by corrupted data by having started a second cluster for a >> short period of time (2 minutes). >> >> I am now trying to repair the data by deleting data that was written >> during or after the parallel run. I hope that fixes the issue and allows to >> recover to a state before the data was corrupted. >> >> On Tuesday, June 30, 2015 at 4:25:56 AM UTC-7, Patrik Nordwall wrote: >>> >>> What version of Akka are you using? >>> Can you run it with debug log level and share those logs? >>> The logging during recovery should tell us what is going on. >>> >>> My guess is that you have ended up with inconsistent data, which may >>> happen if you split the cluster in two separate clusters and the >>> coordinator in both of them are writing to the same persistentId. >>> >>> Regards, >>> Patrik >>> >>> On Fri, Jun 26, 2015 at 9:21 AM, <[email protected]> wrote: >>> >>>> I have a 2 node akka cluster. I am trying to run a persistent actor >>>> with cluster sharding. >>>> When the ShardCoordinator starts up, it is throwing an >>>> IllegalArgumentException. After digging through the source code, I found it >>>> to be a require statement in ClusterSharding.scala >>>> >>>> case ShardHomeAllocated(shard, region) ⇒ >>>> require(regions.contains(region), s"Region $region not >>>> registered: $this") >>>> >>>> >>>> When the ShardCoordinator is started up on app01 (the first node) and >>>> it receives a region from app02 in the event ShardHomeAllocated, the >>>> require statement fails. It also fails when the converse happens. (When >>>> ShardCoordinator is started on app02 and it gets an event with region the >>>> actor from app01). The 'regions' contained in the actor's state must come >>>> from the snapshot which may not contain the region being allocated in the >>>> above event. >>>> >>>> This is what it looks like on app01. Note that the Region actor is from >>>> app02. >>>> >>>> 2015-06-25 14:17:12,017 ERROR akka.actor.OneForOneStrategy - >>>> requirement failed: Region >>>> Actor[akka.tcp://evernym@app02:2551/user/sharding/Subr#-317090222] >>>> not registered: State(Map(8 -> >>>> Actor[akka://evernym/user/sharding/Subr#1934748057]),Map(Actor[akka://evernym/user/sharding/Subr#1934748057] >>>> -> Vector(8)),Set()) >>>> java.lang.IllegalArgumentException: requirement failed: Region Actor[ >>>> akka.tcp://evernym@app02:2551/user/sharding/Subr#-317090222] not >>>> registered: State(Map(8 -> Actor[akka: >>>> >>>> After this exception is thrown, the actor restarts, recovers from >>>> snapshot, applies that event and throws an exception again, going into an >>>> infinite restart loop. >>>> >>>> Any idea why this could be happening? >>>> >>>> >>>> Thanks, >>>> Joseph >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> >>> Patrik Nordwall >>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >>> Twitter: @patriknw >>> >>> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
