That is definitely on my list of concerns. However there are a couple of issues to deal with. The actors are queried by front end users for information and routing the traffic of every user to a single node would be hazardous to stability. Furthermore, the actor could hold state that is anywhere between a couple of KB to several megabytes big (no, that is not avoidable unfortunately) so updating it on one node and sending it to other nodes is not really a viable solution. I am looking at the updates taking up to 3 seconds to process a large batch. I could do the updates with direct messaging to the actors using a wildcard pattern in an actor selector but I would be concerned about the updates being copied. One update could be consumed by any number of actors on the other node and I don't want it copied once per actor.
Yes there is a danger of synch issues with the replicas being out of synch with each other but I am not really sure if there is a viable way to handle that. Even if I updated all on once node and took the serialization hit to send the object to the other nodes for the "read only" actors, there is a danger that the object doesn't arrive and one node is out of synch. One thing that is a limiting factor is that these actors are updated from up to millions of other actors and I wouldn't want them "pulling" state from those other actors on update cycles because that would for sure send the state of the smaller actors many time amongst other nodes. I can't be more specific into the use case because of NDAs but just think the smaller actors have some kind of state but the larger ones have to rank the smaller actors into a ranking system whereby the ranking is affected by the smaller actors. Consider the smaller actors in the millions have a state, lets say that state includes 4 attributes which cooperate in a ranking order. The actors I am replicating maintain the ranking order of the smaller actors and to do that they essentially have to sort the list. The users request some portion of this ranking order but the smaller actors can change state at any moment causing the ranking order to need to be updated. Since the object is hammered by users, driving the load to one machine would be unscalable. Essentially all of our web traffic would end up on one node. So replicating them seemed the right solution, even with the possibility of synch errors. I am open to ideas though. Direct messaging vs pub sub ... if I send a message from node 1 to node 9 and that message is destined for 10 actors on node 9, will it send only once over wire? -- Robert On Thursday, June 2, 2016 at 8:07:25 AM UTC-5, Justin du coeur wrote: > > Hmm. In that case, that's not Cluster Sharding -- but it's also not > common, because it can be challenging to get right (without inconsistencies > between replicas) and I suspect *usually* will result in worse throughput. > (Since you're introducing a lot of PubSub traffic, and replicating the > processing of events.) I'd recommend sanity-checking whether it's actually > cheaper to do it this way when everything is taken into account. > > Assuming so, then no, this is just plain uncommon. But you might want to > take a look at the relatively recent CRDT support, which is the closest > common cognate I can think of to this sort of approach... > > On Wed, Jun 1, 2016 at 9:25 PM, kraythe <[email protected] <javascript:>> > wrote: > >> They are updated from a DistributedPubSub topic. Each of them gets the >> update messages and then recalculates the data they need to maintain >> locally. So essentially they all function as independent actors, isolated >> from each other, knowing nothing about each other. This object is one of >> the most hammered in our system so we need it to scale horizontally and the >> object can possibly hold a HUGE state (which is unavoidable) in the nature >> of 40 megs. So its cheaper to update them all independently. So when an >> update comes in it is published to a Topic that the actors listen to, if >> they care about that particular update, they deal with it. If its an update >> not within their scope, they ignore it. The supervisor just manages the >> objects by key so that you find the right one to interrogate for read ops. >> >> -- Robert >> >> On Wednesday, June 1, 2016 at 5:52:19 PM UTC-5, Ryan Tanner wrote: >>> >>> How are you coordinating state between logically-equal actors on >>> different physical nodes? >>> >>> On Wednesday, June 1, 2016 at 3:24:57 PM UTC-6, kraythe wrote: >>>> >>>> So the reason I didn't think this was cluster sharding is that I >>>> actually want these supervisor actors (and their supervised children) to >>>> be >>>> REPLICATED on every node (they handle user requests). Basically if there >>>> is >>>> an actor with key 10, I want one actor with key 10 per node. I didn't want >>>> the messages getting rerouted to another node. So if I have one of these >>>> actors running on every node how can I do that with sharding? Id imagine I >>>> would have to be fancy with the shard id but i have no idea how. >>>> >>>> -- Robert >>>> >>>> On Wednesday, June 1, 2016 at 3:42:16 PM UTC-5, Konrad Malawski wrote: >>>>> >>>>> >>>>> I was working on a supervisor that lazy creates actors based on some >>>>> key and then will forward messages to that actor. >>>>> >>>>> That's Cluster Sharding :-) >>>>> >>>>> http://doc.akka.io/docs/akka/snapshot/scala/cluster-sharding.html >>>>> >>>>> Technically you can use it on one node too, yeah. >>>>> >>>>> >>>>> Happy hAkking! >>>>> -- >>>>> Konrad `ktoso` Malawski >>>>> Akka <http://akka.io> @ Lightbend <http://lightbend.com> >>>>> >>>>> >>>>> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
