Thanks, Patrik.
I rebuilt my POC with akka release 2.3.2 and re-tested. I was able to
recreate the scenario with in-flight event not being replayed after JVM
crash. Here are some additional parameters and findings pertaining to my
test environment:
- we use channels to pass command from spray-can service end point to the
command handler
- command handler (extends akka.persistence.Processor) confirms receipt of
the event to sender
- command handler sends the event to sharded event processor (extends
akka.persistence.EventsourcedProcessor). This call is deliberately made
w/o use of channel. The communication path between command handler and
event processor is relatively reliable, we do not expect a significant
chance of failure, so we do not use channel between them to preserve
resources.
- event processor persists the event in the event store and continues on
with business logic:
override def receiveCommand: Receive = {
case created: Created ⇒ persist(created)
...
}
The "persist" call stores the event asynchronously. Should the VM crash
while event is being persisted the event is not going to be replayed when
the processor is restored (VM is restarted).
To refine my original question: is use of channels the only akka sanctioned
method of assuring successful replay of event in case of processor failure?
Thanks in advance!
~g
On Friday, April 25, 2014 10:01:54 AM UTC-4, Patrik Nordwall wrote:
>
> First of all, use latest stable release: 2.3.2
> /Patrik
>
>
> On Fri, Apr 25, 2014 at 2:45 PM, George Wilk
> <[email protected]<javascript:>
> > wrote:
>
>> Dependencies:
>> - akka-persistence-experimental 2.3.0
>> - akka-actor 2.3.0
>> - akka-contrib 2.3.0
>>
>> We are building large CQRS system utilizing Akka-persistence with cluster
>> sharding in akka-contrib,
>> deploying multiple EventsourcedProcessors distributed on several cluster
>> nodes.
>>
>> As part of application resilience testing we are running the following
>> test scenario:
>>
>> - 2 akka nodes (node_1 and node_2) in 2 individual VMs deployed on 2
>> separate servers
>> - bring the cluster up and let it acquiesce: first node_1 then node_2
>> - start generating event messages
>> - while under load, shut down one of the VMs - node_2 (non-seed node)
>> - observe cluster state: several in-process event messages being
>> processed by the processor
>> on shut down VM are incomplete/unprocessed. This is expected as result
>> of VM crash.
>> - unprocessed events are not replayed while node_2 is down.
>> - restart node_2. Expected to see unprocessed events to be replayed, but
>> it does not happen.
>> - restart cluster in the same order as in original setup: first node_1,
>> then node_2
>> - observe cluster state: unprocessed messages have been replayed
>>
>> Questions:
>> 1) should we expect that once node failure (processor failure) is
>> detected by akka cluster
>> the unprocessed events will be replayed while the failed node (node_2) is
>> still down?
>> 2) if answer to 1) is NO, should we expect that unprocessed events will
>> be replayed once
>> failed node (node_2) comes back online?
>> 3) is there a difference in cluster behavior when the failed node is a
>> seed node vs. non-seed
>> node?
>>
>> Thanks in advance!
>> ~g
>>
>> --
>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>> >>>>>>>>>> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To post to this group, send email to [email protected]<javascript:>
>> .
>> Visit this group at http://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
>
> Patrik Nordwall
> Typesafe <http://typesafe.com/> - Reactive apps on the JVM
> Twitter: @patriknw
> JOIN US. REGISTER TODAY! <http://www.scaladays.org/>
> Scala <http://www.scaladays.org/>
> Days <http://www.scaladays.org/>
> June 16th-18th, <http://www.scaladays.org/>
> Berlin <http://www.scaladays.org/>
>
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
