Finally we were able to identify the root cause - I've opened an issue that describes the problem in details: https://github.com/akka/akka/issues/22768
On Wednesday, April 12, 2017 at 12:56:11 PM UTC+3, Serg wrote: > > @Patrick - > akka-persistence-cassandra, 0.22 > > @Arnout - > "How did you verify this is the case" - looking at Shard and PA state in > debug mode > > "Is there any chance you could extract some running example code" - We'll > try to extract an isolated code example. Right now scenario to reproduce > the problem involves sending 200+ messages to multiple PA instances to get > PA one instance stuck. > > > On Wednesday, April 12, 2017 at 11:40:49 AM UTC+3, Patrik Nordwall wrote: >> >> What persistence (journal/snapshot) plugin do you use? >> ons 12 apr. 2017 kl. 06:43 skrev Arnout Engelen <[email protected] >> >: >> >>> Hello Serg, >>> >>> '2' is actually working as designed: if the actor does not crash, the >>> shard has no way of knowing whether to expect a response (and when). >>> >>> Would an actor crash, then the shard handle this because it is the >>> actors' supervisor. When it terminates normally it also gets notified >>> because it watches the actor. >>> >>> I agree a persistent actor should never get into a state where it does >>> not accept any new messages but does not crash either. How did you verify >>> this is the case? Is there any chance you could extract some running >>> example code that also exhibits this problem? >>> >>> I don't think I have a workaround for you, at least not until we >>> pointpoint what exactly is going on. >>> >>> >>> Kind regards, >>> >>> Arnout >>> >>> On Mon, Apr 10, 2017 at 10:31 AM, Serg <[email protected]> wrote: >>> >>>> Hi there, >>>> >>>> Faced the following problem while using Cluster Sharding. There are >>>> actually 2 problems to it: >>>> >>>> 1. Persistent Actor stuck for some reason in 'Persisting events' state >>>> forever. No errors or issues in the log. Even in the case of persistent >>>> storage failure one would expect Persistence Actor to fail, not to stuck >>>> >>>> 2. They way Shard is forwarding messages it doesn't take into account >>>> that target persistent Actor could be in inconsistent state. See >>>> https://github.com/akka/akka/blob/master/akka-cluster-sharding/src/main/scala/akka/cluster/sharding/Shard.scala >>>> >>>> : 296 >>>> >>>> case Some(actor) ⇒ actor.tell(payload, snd) case None ⇒ >>>> getEntity(id).tell(payload, snd) >>>> >>>> >>>> As the result Persistence Actor is just silently dropping messages >>>> while Shard hosting that Actor thinks everything is all right. >>>> >>>> Any ideas and suggestions on >>>> 1. Why Actor could stuck in 'Persisting events' state? >>>> 2. What would be the way to identify situation when Shard is dropping >>>> messages? >>>> 3. What are potential workarounds here? >>>> >>>> are appreciated >>>> >>>> (P.S. We are running Akka 2.4.16) >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Arnout Engelen >>> *Senior Software Engineer* >>> E: [email protected] >>> T: https://twitter.com/raboofje >>> >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: >>> http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
