Thank you all for the insightful comments. It turns out that the
unprocessed event was being offered to the processor for replay following
node failure, but our logic verifying event sequence numbers (kicks) was
rejecting it due to environmental setup of the test itself. We are
revisiting our handling of persist-acknowledgment in current design based
on Martin's comments.
Best,
~george
On Wednesday, April 30, 2014 1:45:37 AM UTC-4, Martin Krasser wrote:
>
>
> On 29.04.14 19:01, George Wilk wrote:
>
> Thanks, Patrik.
>
> I rebuilt my POC with akka release 2.3.2 and re-tested. I was able to
> recreate the scenario with in-flight event not being replayed after JVM
> crash. Here are some additional parameters and findings pertaining to my
> test environment:
> - we use channels to pass command from spray-can service end point to the
> command handler
> - command handler (extends akka.persistence.Processor) confirms receipt of
> the event to sender
> - command handler sends the event to sharded event processor (extends
> akka.persistence.EventsourcedProcessor). This call is deliberately made
> w/o use of channel. The communication path between command handler and
> event processor is relatively reliable, we do not expect a significant
> chance of failure, so we do not use channel between them to preserve
> resources.
> - event processor persists the event in the event store and continues on
> with business logic:
>
> override def receiveCommand: Receive = {
> case created: Created ⇒ persist(created)
> ...
> }
>
> The "persist" call stores the event asynchronously. Should the VM crash
> while event is being persisted the event is not going to be replayed when
> the processor is restored (VM is restarted).
>
> To refine my original question: is use of channels the only akka
> sanctioned method of assuring successful replay of event in case of
> processor failure?
>
>
> No, as Endre already said, use an event handler when calling persist(). In
> this event handler generate a reply to the original sender. When the event
> handler is called by the EventsourcedProcessor, you can be sure that the
> event has been successfully persisted. Following this strategy, there's no
> need to make the command handler a Processor, nor do you need a channel in
> front of that Processor. Even if you'd use a persistent channel, you'd need
> to wait for a persist-acknowledgement from the channel in order to be sure
> that the command has been persisted (and return it to the initial sender).
> This is equivalent to sending a reply from within the event handler. Hence,
> you can directly route from spray to the EventsourcedProcessors (via some
> non-persistent intermediate actors, if needed).
>
>
> Thanks in advance!
> ~g
>
> On Friday, April 25, 2014 10:01:54 AM UTC-4, Patrik Nordwall wrote:
>>
>> First of all, use latest stable release: 2.3.2
>> /Patrik
>>
>>
>> On Fri, Apr 25, 2014 at 2:45 PM, George Wilk <[email protected]>wrote:
>>
>>> Dependencies:
>>> - akka-persistence-experimental 2.3.0
>>> - akka-actor 2.3.0
>>> - akka-contrib 2.3.0
>>>
>>> We are building large CQRS system utilizing Akka-persistence with
>>> cluster sharding in akka-contrib,
>>> deploying multiple EventsourcedProcessors distributed on several cluster
>>> nodes.
>>>
>>> As part of application resilience testing we are running the following
>>> test scenario:
>>>
>>> - 2 akka nodes (node_1 and node_2) in 2 individual VMs deployed on 2
>>> separate servers
>>> - bring the cluster up and let it acquiesce: first node_1 then node_2
>>> - start generating event messages
>>> - while under load, shut down one of the VMs - node_2 (non-seed node)
>>> - observe cluster state: several in-process event messages being
>>> processed by the processor
>>> on shut down VM are incomplete/unprocessed. This is expected as result
>>> of VM crash.
>>> - unprocessed events are not replayed while node_2 is down.
>>> - restart node_2. Expected to see unprocessed events to be replayed,
>>> but it does not happen.
>>> - restart cluster in the same order as in original setup: first node_1,
>>> then node_2
>>> - observe cluster state: unprocessed messages have been replayed
>>>
>>> Questions:
>>> 1) should we expect that once node failure (processor failure) is
>>> detected by akka cluster
>>> the unprocessed events will be replayed while the failed node (node_2)
>>> is still down?
>>> 2) if answer to 1) is NO, should we expect that unprocessed events will
>>> be replayed once
>>> failed node (node_2) comes back online?
>>> 3) is there a difference in cluster behavior when the failed node is a
>>> seed node vs. non-seed
>>> node?
>>>
>>> Thanks in advance!
>>> ~g
>>> --
>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>> >>>>>>>>>> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >>>>>>>>>> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at http://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>>
>> Patrik Nordwall
>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM
>> Twitter: @patriknw
>> JOIN US. REGISTER TODAY! <http://www.scaladays.org/>
>> Scala <http://www.scaladays.org/>
>> Days <http://www.scaladays.org/>
>> June 16th-18th, <http://www.scaladays.org/>
>> Berlin <http://www.scaladays.org/>
>>
> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <javascript:>.
> To post to this group, send email to [email protected]<javascript:>
> .
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> Martin Krasser
>
> blog: http://krasserm.blogspot.com
> code: http://github.com/krasserm
> twitter: http://twitter.com/mrt1nz
>
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
