Hi Danny, what you are seeing is normal behavior. Since sending messages is at-most-once-delivery, there is no implicit guarantee that all messages that are being sent are going to be delivered to the destination. Akka's tries hard that messages are not lost, but if someone pulls the plug on the JVM there is no default mechanisms to make sure no messages are lost.
You have noticed correctly that if you are leaving the Cluster gracefully, then no messages are lost. There is still some change the messages could be lost in this situation, but it is slim. You can also shutdown ShardRegion and only then leave the cluster (feature in Akka 2.4). This gives Akka even more chance to ensure that as little messages are lost. All this said, if your usecase really needs stronger guarantee of message delivery, you need to persist messages that are inflight and resend messages that are not acknowledged. You could use AtLeastOnceDelivery <http://doc.akka.io/docs/akka/2.3.12/scala/persistence.html#At-Least-Once_Delivery> trait or a custom solution to achieve this. On Fri, Jul 10, 2015 at 6:12 PM, Danny Lesnik <[email protected]> wrote: > I've created POC with three nodes in the cluster and have the following > scenario: > > I have modified Counter scala example from Cluster Sharding docs. and have > two shards one of odd number and second for even number and persistent > shard Actor Counter which sums messages it receives and persists its > state. > > I have a singleton actor which sends odd and even numbers as messages > every second to shard region. > > So I have the following: > > - First Node has a singleton and and have a shard for even numbers, it > receives 2,4,6,8,etc... Counter actor receives even numbers sums and > persists it. > - Second Node has a shard for an odd numbers, an it receive 1,3,5,7, > etc... Counter actor receives odd numbers sums and persists it. > - Third Node is just on standby and does nothing. > > After about two minutes I kill JVM on node #2. as the result after about > ten seconds Node #3 creates a shard actor for the odd number and starts to > receive messages. However the problem is that all messages send during > those 10 seconds are lost. For example, if 135 is the last message that > odd shard received the next message that would be received by node #3 will > be 147. So I have 10 messages lost. > > Is it normal behavior? is there any option to avoid losing messages when > nod with a shard fails? > > P.S: If after 2 minutes I shutting down node #2 gracefully, by leaving a > cluster using Cluster.leave command waiting for Node Removed Event and then > shutdown ActorSystem, Node #3 immediately creates shard for odd numbers and > no message are lost. > > Can anybody help? > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Martynas Mickevičius Typesafe <http://typesafe.com/> – Reactive <http://www.reactivemanifesto.org/> Apps on the JVM -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
