Hello! I'm hoping the wisdom of the Akka community can give me some
insight! I'm trying to use a ClusterClient to invoke an actor in a cluster
(v. 2.4.17), but the messages are always timing out. Here's my setup:
a) An Akka cluster running on AWS ECS (where each Akka node is a Dockerized
app)
b) 3 Akka nodes designated as seed nodes (role = "seed"), one running on
each of three ECS hosts
c) Each of these seed nodes also has a (manually instantiated)
DistributedPubSubMediator and ClusterReceptionist running on it
d) 1 Akka node (role = "test1") which provides a pool of "StatsWorker"
actors ("/user/statsWorkerPool")
e) 1 Akka node (role = "test2") which provides a "StatsService" actor
("/user/statsService") which (in part) leverages /user/statsWorkerPool
f) This /user/statsService is registered with the ClusterClientReceptionist
at the startup of e)
This, so far, all appears to work. I have a 5 node Akka cluster, and if I
send a message to /user/statsService from the same node in e) I get the
answer back I expect (leveraging the StatsWorker actors on the node in (d)
as well).
Now I'm trying to access /user/statsService from outside the cluster. I
have a 6th Docker container running in the same ECS environment (for
testing) which attempts to use a ClusterClient to access the cluster
created above. The client app defines its initial contacts using the three
seed nodes (which are running the receptionist) like so:
akka.tcp://[email protected]:2551/user/receptionist
akka.tcp://[email protected]:2551/user/receptionist
akka.tcp://[email protected]:2551/user/receptionist
(It's "/user/receptionist" because I'm manually instantiating the
receptionist actors -- see below for why)
The client app seems to find the cluster and establish the relationship:
[DEBUG] [03/14/2017 19:39:15.304]
[RestApp-akka.actor.default-dispatcher-18]
[akka.tcp://[email protected]:2552/user/client] Sending GetContacts to
[ActorSelection[Anchor(akka.tcp://[email protected]:2551/),
Path(/user/receptionist)],ActorSelection[Anchor(akka.tcp://[email protected]:2551/),
Path(/user/receptionist)],ActorSelection[Anchor(akka.tcp://[email protected]:2551/),
Path(/user/receptionist)]]
[DEBUG] [03/14/2017 19:39:15.630]
[RestApp-akka.remote.default-remote-dispatcher-16]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.32.90%3A2551-2/endpointWriter]
Associated [akka.tcp://[email protected]:2552] ->
[akka.tcp://[email protected]:2551]
[DEBUG] [03/14/2017 19:39:15.636]
[RestApp-akka.remote.default-remote-dispatcher-8]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.30.80%3A2551-0/endpointWriter]
Associated [akka.tcp://[email protected]:2552] ->
[akka.tcp://[email protected]:2551]
etc...
The client attempts to "ask" the "/user/statsService" using this format:
ActorRef client = actorSystem.actorFor("/user/client"); // The actor
established at client startup, using the ClusterClient w/ initial contacts
Future<Object> future = Patterns.ask(client, new
ClusterClient.Send("/user/statsService", new StatsMessages.StatsJob("hello
world")), timeout);
When this request is made, the client disassociates from the cluster, and
the "ask" times out. The following logs are generated on the client:
[DEBUG] [03/14/2017 19:40:12.415]
[RestApp-akka.remote.default-remote-dispatcher-16]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.61.224%3A2551-1/endpointWriter]
Disassociated [akka.tcp://[email protected]:2552] ->
[akka.tcp://[email protected]:2551]
[WARN] [03/14/2017 19:40:12.422]
[RestApp-akka.remote.default-remote-dispatcher-16]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.61.224%3A2551-1]
Association with remote system [akka.tcp://[email protected]:2551]
has failed, address is now gated for [5000] ms. Reason: [Disassociated]
[DEBUG] [03/14/2017 19:40:12.438]
[RestApp-akka.remote.default-remote-dispatcher-8]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.61.224%3A2551-1/endpointWriter]
Disassociated [akka.tcp://[email protected]:2552] ->
[akka.tcp://[email protected]:2551]
[DEBUG] [03/14/2017 19:40:19.296]
[RestApp-akka.remote.default-remote-dispatcher-16]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.61.224%3A2551-5/endpointWriter]
Associated [akka.tcp://[email protected]:2552] ->
[akka.tcp://[email protected]:2551]
[DEBUG] [03/14/2017 19:40:19.298]
[RestApp-akka.remote.default-remote-dispatcher-16]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.61.224%3A2551-5/endpointWriter]
Drained buffer with maxWriteCount: 50, fullBackoffCount: 1,
smallBackoffCount: 0, noBackoffCount: 0 , adaptiveBackoff: 1000
[my own timeout error message displayed here]
So I have several issues. Most importantly, I'm unable to get a response
back from the /user/statsService actor from outside the Akka cluster -- it
always times out. Additionally, once the above "disassociation" scenario
happens and I restart the client app, it won't "re-associate" with the
cluster (it just continually sends 'GetContacts'). It takes a restart of
the entire Akka cluster for the client to successfully associate again.
Finally, I manually instantiate the mediator and receptionist on each seed
node because when using the ClusterClientReceptionist extension on the seed
nodes, the client continually sends "GetContacts" and never associates with
the cluster.
If you've read this far, you're a brave soul, and I really appreciate it!
Any help -- even some suggested troubleshooting steps or where to look
next -- is much appreciated. Right now I'm pretty much at a dead end.
Thank you!
- Paul
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
