Hi, We have a setup of an akka cluster with the following: seed node, a master node, and worker nodes. Workers will broadcast a request work message that the master node respond to. We deployed this setup in Google Kubernetes environment. It usually works fine. workers sometimes run out of memory but docker will restart the pod and they usually are able to reconnect to the akka cluster. We recently had an incident where all the worker nodes were busy, which is normal, and all of a sudden the seed node cpu shot up and stayed up. At that point we saw network traffic for seed, master, and worker nodes drop to 0 at the same time (Datadog graphs). We had to restart the components to fix the problem. Once restarted they were fine and continued to do the job. The seed node memory usage was close to the limit but docker did not restart the pod.
The seed node has very little logic, it's main purpose is to serve as an entry point to form the akka cluster. There are 40+ workers and 1 master. So I have the following questions: 1. Correct me if I'm wrong but when a cluster is formed the seed node can disappear and the cluster will continue to function. The impact in this scenario is no new workers can join the existing cluster. 2. The DataDog graphs do indicate when the seed node had high cpu load it seems to stop the worker's message from reaching the master. Is that what is happening? Please note the workers are broadcasting, not sending messages directly to the master. 3. With the number of workers and the frequency of broadcast messages is it possible to be overloading the nodes in the cluster with messages? Thanks -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
