I ended up solving my problem and thought I'd post back here in case
someone stumbles across it looking for something similar. I actually had
two issues:
First, the reason I couldn't use the ClusterClientReceptionist extension is
because the application.conf on the seed/receptionist nodes looked like
this:
akka {
actor {
...
}
remote {
...
}
cluster.roles = ["seed"]
akka.extensions = ["akka.cluster.client.ClusterClientReceptionist"]
akka.cluster.client.receptionist {
...
}
}
These last two entries are, of course, wrong since they already appear
inside "akka". Changing them to 'extensions' and
'cluster.client.receptionist' respectively resolved that issue. As I was
working through this, I came across the Configuration Checker and added
"akka.diagnostics.checker.fail-on-warning = on", only to learn later that
it's part of a paid extension. The error was definitely an oversight on my
part, but having some sort of very rudimentary syntax checking in the
default Akka platform would be helpful for new users in cases like this.
(Also, an "upsell" error message when adding an akka.diagnostics
configuration entry would be helpful too :-)
The second issue, which I believe caused the Disassociation, was because
the node running the Receptionist didn't have access to the Java class
containing the message being sent to the StatsService actor in the cluster.
The flow looks like this:
[ClusterClient] -> [Receptionist running on a Seed Node] ->
[/user/statsService running on a different node]
Both the ClusterClient and the node running /user/statsService were
deployed with the appropriate Java code to define the StatsMessages
classes. However, the Receptionist/Seed node was not, and it would throw
an error/stack trace whenever the ClusterClient attempted to send a
StatsMessages.StatsJob message through it. Adding these classes to the
seed node (for now) solved the problem. This seems to indicate that
receptionists should be run in close coordination with (if not directly on)
nodes exposing a particular actor outside the cluster.
Anyway, I hope those solutions help someone else!
Thanks,
Paul
On Tuesday, March 14, 2017 at 5:27:43 PM UTC-4, Paul O'Fallon wrote:
>
> Hello! I'm hoping the wisdom of the Akka community can give me some
> insight! I'm trying to use a ClusterClient to invoke an actor in a cluster
> (v. 2.4.17), but the messages are always timing out. Here's my setup:
>
> a) An Akka cluster running on AWS ECS (where each Akka node is a
> Dockerized app)
> b) 3 Akka nodes designated as seed nodes (role = "seed"), one running on
> each of three ECS hosts
> c) Each of these seed nodes also has a (manually instantiated)
> DistributedPubSubMediator and ClusterReceptionist running on it
> d) 1 Akka node (role = "test1") which provides a pool of "StatsWorker"
> actors ("/user/statsWorkerPool")
> e) 1 Akka node (role = "test2") which provides a "StatsService" actor
> ("/user/statsService") which (in part) leverages /user/statsWorkerPool
> f) This /user/statsService is registered with the
> ClusterClientReceptionist at the startup of e)
>
> This, so far, all appears to work. I have a 5 node Akka cluster, and if I
> send a message to /user/statsService from the same node in e) I get the
> answer back I expect (leveraging the StatsWorker actors on the node in (d)
> as well).
>
> Now I'm trying to access /user/statsService from outside the cluster. I
> have a 6th Docker container running in the same ECS environment (for
> testing) which attempts to use a ClusterClient to access the cluster
> created above. The client app defines its initial contacts using the three
> seed nodes (which are running the receptionist) like so:
>
> akka.tcp://[email protected]:2551/user/receptionist
> akka.tcp://[email protected]:2551/user/receptionist
> akka.tcp://[email protected]:2551/user/receptionist
>
>
> (It's "/user/receptionist" because I'm manually instantiating the
> receptionist actors -- see below for why)
>
> The client app seems to find the cluster and establish the relationship:
>
> [DEBUG] [03/14/2017 19:39:15.304]
> [RestApp-akka.actor.default-dispatcher-18] [akka.tcp://
> [email protected]:2552/user/client] Sending GetContacts to
> [ActorSelection[Anchor(akka.tcp://[email protected]:2551/),
> Path(/user/receptionist)],ActorSelection[Anchor(akka.tcp://
> [email protected]:2551/),
> Path(/user/receptionist)],ActorSelection[Anchor(akka.tcp://
> [email protected]:2551/), Path(/user/receptionist)]]
> [DEBUG] [03/14/2017 19:39:15.630]
> [RestApp-akka.remote.default-remote-dispatcher-16] [akka.tcp://
> [email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.32.90%3A2551-2/endpointWriter]
>
> Associated [akka.tcp://[email protected]:2552] -> [akka.tcp://
> [email protected]:2551]
> [DEBUG] [03/14/2017 19:39:15.636]
> [RestApp-akka.remote.default-remote-dispatcher-8] [akka.tcp://
> [email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.30.80%3A2551-0/endpointWriter]
>
> Associated [akka.tcp://[email protected]:2552] -> [akka.tcp://
> [email protected]:2551]
> etc...
>
> The client attempts to "ask" the "/user/statsService" using this format:
>
> ActorRef client = actorSystem.actorFor("/user/client"); // The actor
> established at client startup, using the ClusterClient w/ initial contacts
> Future<Object> future = Patterns.ask(client, new
> ClusterClient.Send("/user/statsService", new StatsMessages.StatsJob("hello
> world")), timeout);
>
> When this request is made, the client disassociates from the cluster, and
> the "ask" times out. The following logs are generated on the client:
>
> [DEBUG] [03/14/2017 19:40:12.415]
> [RestApp-akka.remote.default-remote-dispatcher-16] [akka.tcp://
> [email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.61.224%3A2551-1/endpointWriter]
>
> Disassociated [akka.tcp://[email protected]:2552] -> [akka.tcp://
> [email protected]:2551]
> [WARN] [03/14/2017 19:40:12.422]
> [RestApp-akka.remote.default-remote-dispatcher-16] [akka.tcp://
> [email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.61.224%3A2551-1]
>
> Association with remote system [akka.tcp://[email protected]:2551]
> has failed, address is now gated for [5000] ms. Reason: [Disassociated]
> [DEBUG] [03/14/2017 19:40:12.438]
> [RestApp-akka.remote.default-remote-dispatcher-8] [akka.tcp://
> [email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.61.224%3A2551-1/endpointWriter]
>
> Disassociated [akka.tcp://[email protected]:2552] -> [akka.tcp://
> [email protected]:2551]
> [DEBUG] [03/14/2017 19:40:19.296]
> [RestApp-akka.remote.default-remote-dispatcher-16] [akka.tcp://
> [email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.61.224%3A2551-5/endpointWriter]
>
> Associated [akka.tcp://[email protected]:2552] -> [akka.tcp://
> [email protected]:2551]
> [DEBUG] [03/14/2017 19:40:19.298]
> [RestApp-akka.remote.default-remote-dispatcher-16] [akka.tcp://
> [email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterrific%40172.31.61.224%3A2551-5/endpointWriter]
>
> Drained buffer with maxWriteCount: 50, fullBackoffCount: 1,
> smallBackoffCount: 0, noBackoffCount: 0 , adaptiveBackoff: 1000
> [my own timeout error message displayed here]
>
> So I have several issues. Most importantly, I'm unable to get a response
> back from the /user/statsService actor from outside the Akka cluster -- it
> always times out. Additionally, once the above "disassociation" scenario
> happens and I restart the client app, it won't "re-associate" with the
> cluster (it just continually sends 'GetContacts'). It takes a restart of
> the entire Akka cluster for the client to successfully associate again.
> Finally, I manually instantiate the mediator and receptionist on each seed
> node because when using the ClusterClientReceptionist extension on the seed
> nodes, the client continually sends "GetContacts" and never associates with
> the cluster.
>
> If you've read this far, you're a brave soul, and I really appreciate it!
> Any help -- even some suggested troubleshooting steps or where to look
> next -- is much appreciated. Right now I'm pretty much at a dead end.
>
> Thank you!
>
> - Paul
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
