On 29.04.14 19:01, George Wilk wrote:
Thanks, Patrik.
I rebuilt my POC with akka release 2.3.2 and re-tested. I was able to
recreate the scenario with in-flight event not being replayed after
JVM crash. Here are some additional parameters and findings
pertaining to my test environment:
- we use channels to pass command from spray-can service end point to
the command handler
- command handler (extends akka.persistence.Processor) confirms
receipt of the event to sender
- command handler sends the event to sharded event processor (extends
akka.persistence.EventsourcedProcessor). This call is deliberately
made w/o use of channel. The communication path between command
handler and event processor is relatively reliable, we do not expect a
significant chance of failure, so we do not use channel between them
to preserve resources.
- event processor persists the event in the event store and continues
on with business logic:
override def receiveCommand: Receive = {
case created: Created ⇒ persist(created)
...
}
The "persist" call stores the event asynchronously. Should the VM
crash while event is being persisted the event is not going to be
replayed when the processor is restored (VM is restarted).
To refine my original question: is use of channels the only akka
sanctioned method of assuring successful replay of event in case of
processor failure?
No, as Endre already said, use an event handler when calling persist().
In this event handler generate a reply to the original sender. When the
event handler is called by the EventsourcedProcessor, you can be sure
that the event has been successfully persisted. Following this strategy,
there's no need to make the command handler a Processor, nor do you need
a channel in front of that Processor. Even if you'd use a persistent
channel, you'd need to wait for a persist-acknowledgement from the
channel in order to be sure that the command has been persisted (and
return it to the initial sender). This is equivalent to sending a reply
from within the event handler. Hence, you can directly route from spray
to the EventsourcedProcessors (via some non-persistent intermediate
actors, if needed).
Thanks in advance!
~g
On Friday, April 25, 2014 10:01:54 AM UTC-4, Patrik Nordwall wrote:
First of all, use latest stable release: 2.3.2
/Patrik
On Fri, Apr 25, 2014 at 2:45 PM, George Wilk
<[email protected] <javascript:>> wrote:
Dependencies:
- akka-persistence-experimental 2.3.0
- akka-actor 2.3.0
- akka-contrib 2.3.0
We are building large CQRS system utilizing Akka-persistence
with cluster sharding in akka-contrib,
deploying multiple EventsourcedProcessors distributed on
several cluster nodes.
As part of application resilience testing we are running the
following test scenario:
- 2 akka nodes (node_1 and node_2) in 2 individual VMs
deployed on 2 separate servers
- bring the cluster up and let it acquiesce: first node_1 then
node_2
- start generating event messages
- while under load, shut down one of the VMs - node_2
(non-seed node)
- observe cluster state: several in-process event messages
being processed by the processor
on shut down VM are incomplete/unprocessed. This is expected
as result of VM crash.
- unprocessed events are not replayed while node_2 is down.
- restart node_2. Expected to see unprocessed events to be
replayed, but it does not happen.
- restart cluster in the same order as in original setup:
first node_1, then node_2
- observe cluster state: unprocessed messages have been replayed
Questions:
1) should we expect that once node failure (processor failure)
is detected by akka cluster
the unprocessed events will be replayed while the failed node
(node_2) is still down?
2) if answer to 1) is NO, should we expect that unprocessed
events will be replayed once
failed node (node_2) comes back online?
3) is there a difference in cluster behavior when the failed
node is a seed node vs. non-seed
node?
Thanks in advance!
~g
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
<http://doc.akka.io/docs/akka/current/additional/faq.html>
>>>>>>>>>> Search the archives:
https://groups.google.com/group/akka-user
<https://groups.google.com/group/akka-user>
---
You received this message because you are subscribed to the
Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to [email protected] <javascript:>.
To post to this group, send email to [email protected]
<javascript:>.
Visit this group at http://groups.google.com/group/akka-user
<http://groups.google.com/group/akka-user>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
Patrik Nordwall
Typesafe <http://typesafe.com/> - Reactive apps on the JVM
Twitter: @patriknw
JOIN US. REGISTER TODAY! <http://www.scaladays.org/>
Scala <http://www.scaladays.org/>
Days <http://www.scaladays.org/>
June 16th-18th, <http://www.scaladays.org/>
Berlin <http://www.scaladays.org/>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Martin Krasser
blog: http://krasserm.blogspot.com
code: http://github.com/krasserm
twitter: http://twitter.com/mrt1nz
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
