I have an odd bug in production that I can't figure out and am wondering if
it sounds familiar to anyone.
Background:
I have 3 components: UI, CommandService, and OptimizerAdapter.
The seed nodes are as follows:
UI list: CommandService nodes, UI nodes
OptimizerAdapter list: CommandService nodes, OptimizerAdapter nodes
CommandService list: CommandService nodes, OptimizerAdapter nodes, UI nodes
We do this to make the Command Service the "candle stick" of sorts that
ties the cluster together.
Physically, the OptimizerAdapter and CommandService are colocated, while
the UI lives on a different set of VMs
Messaging flow:
1. User inputs a request to the UI. UI creates an Actor to handle the
specific request
2. UI sends the request on the pub sub mediator with the actor created
in step 1 as the sender
3. OptimizerAdapter is listening on the mediator topic and picks up the
request
4. OptimizerAdapter creates an Actor to handle this specific request so
it is free to listen for more requests from the mediator
5. The Actor from Step 4 sends some commands to the CommandService, and
waits until it hears back from it to move to step 6
6. The Actor from Step 4 then kicks off an optimization run (via a
message published to JMS)
7. The Actor from Step 4 listens on a JMS topic and relays status
updates to the original sender from step 2
1. This sender was passed in at actor creation
8. The UI sender updates the UI as necessary as it receives status
updates from step 7
9. When the optimization run is complete, the actor from step 4 fires
off a COMPLETE message to the UI and kills himself
10. When the UI receives a complete or fail, it kills its handler actor
This flow seems to work fine about half the time. If I kick off 2
concurrent requests to the UI, we seem ok. If I do more than that, Step 8
never happens. My OptimizerAdapter is able to receive the request from the
UI, but the UI never receives any status updates in its handler despite my
logs showing that the OptimizerAdapter is calling .tell on the ActorRef
passed in by getSender().
Following the Actor per request model seemed appropriate here, but I am
unsure why the responses to the UI seem to never happen. I am not seeing
any deadletters logs or anything. Just a log indicating that I called
originalSender.tell and nothing on the other end (UI logs for the handler)
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
