Hi George,
On Tue, Apr 29, 2014 at 7:01 PM, George Wilk <[email protected]>wrote: > Thanks, Patrik. > > I rebuilt my POC with akka release 2.3.2 and re-tested. I was able to > recreate the scenario with in-flight event not being replayed after JVM > crash. Here are some additional parameters and findings pertaining to my > test environment: > If you have a small piece of code that reproduces the problem you should post it, it helps us to see where the problem might come from. > - we use channels to pass command from spray-can service end point to the > command handler > Do you use persistent channels or simple channels? > - command handler (extends akka.persistence.Processor) confirms receipt of > the event to sender > - command handler sends the event to sharded event processor (extends > akka.persistence.EventsourcedProcessor). This call is deliberately made > w/o use of channel. The communication path between command handler and > event processor is relatively reliable, we do not expect a significant > chance of failure, so we do not use channel between them to preserve > resources. > - event processor persists the event in the event store and continues on > with business logic: > > override def receiveCommand: Receive = { > case created: Created ⇒ persist(created) > ... > } > > The "persist" call stores the event asynchronously. Should the VM crash > while event is being persisted the event is not going to be replayed when > the processor is restored (VM is restarted). > This is true, but persist() takes a second parameter (see the examples: http://doc.akka.io/docs/akka/2.3.2/scala/persistence.html#Event_sourcing) that takes a callback which is called when persisting has been successful. > > To refine my original question: is use of channels the only akka > sanctioned method of assuring successful replay of event in case of > processor failure? > No, you can introduce confirmation messages yourself, then it is the senders responsibility to wait for acknowledgements and resend if necessary. The sender should obviously persist the fact that it is trying to deliver, and after confirmation it should persist the event that it has been successful. That said, channels are possibly the easiest and safest way to do it. -Endre > > Thanks in advance! > ~g > > On Friday, April 25, 2014 10:01:54 AM UTC-4, Patrik Nordwall wrote: > >> First of all, use latest stable release: 2.3.2 >> /Patrik >> >> >> On Fri, Apr 25, 2014 at 2:45 PM, George Wilk <[email protected]>wrote: >> >>> Dependencies: >>> - akka-persistence-experimental 2.3.0 >>> - akka-actor 2.3.0 >>> - akka-contrib 2.3.0 >>> >>> We are building large CQRS system utilizing Akka-persistence with >>> cluster sharding in akka-contrib, >>> deploying multiple EventsourcedProcessors distributed on several cluster >>> nodes. >>> >>> As part of application resilience testing we are running the following >>> test scenario: >>> >>> - 2 akka nodes (node_1 and node_2) in 2 individual VMs deployed on 2 >>> separate servers >>> - bring the cluster up and let it acquiesce: first node_1 then node_2 >>> - start generating event messages >>> - while under load, shut down one of the VMs - node_2 (non-seed node) >>> - observe cluster state: several in-process event messages being >>> processed by the processor >>> on shut down VM are incomplete/unprocessed. This is expected as result >>> of VM crash. >>> - unprocessed events are not replayed while node_2 is down. >>> - restart node_2. Expected to see unprocessed events to be replayed, >>> but it does not happen. >>> - restart cluster in the same order as in original setup: first node_1, >>> then node_2 >>> - observe cluster state: unprocessed messages have been replayed >>> >>> Questions: >>> 1) should we expect that once node failure (processor failure) is >>> detected by akka cluster >>> the unprocessed events will be replayed while the failed node (node_2) >>> is still down? >>> 2) if answer to 1) is NO, should we expect that unprocessed events will >>> be replayed once >>> failed node (node_2) comes back online? >>> 3) is there a difference in cluster behavior when the failed node is a >>> seed node vs. non-seed >>> node? >>> >>> Thanks in advance! >>> ~g >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>> current/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> >> Patrik Nordwall >> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >> Twitter: @patriknw >> JOIN US. REGISTER TODAY! <http://www.scaladays.org/> >> Scala <http://www.scaladays.org/> >> Days <http://www.scaladays.org/> >> June 16th-18th, <http://www.scaladays.org/> >> Berlin <http://www.scaladays.org/> >> >> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
