Always thought provoking response. The issue is that I have a number of entities that are looking for certain events and updating themselved based on those events. If they change then they post another message to a topic indicating they were updated. There is another entity that integrates those changes into a analytics calculation. So the entities I am trying to inflate are A and there are millions of them. If they change then they inform all the Bs that care. Multiple Bs care about potentially the same A. I thought about doing all the updating on every B but I was concerned this would cause the analytics calculation in B to take that much longer. Already its evaluating data from millions of As but currently its just sorting. If I integrate the events into Bs and update the objects from within the B then the time could tripple. So the idea was to inflate all the As that feed the Bs and then have them detect updates and feed a topic for the Bs. Of course that requires all the As participating to be alive to get the distributed pub sub messages.
For now I just power through sending all 1m As an init message but the implementation bothers me. On Friday, June 10, 2016 at 2:28:46 PM UTC-5, Justin du coeur wrote: > > Ahhh -- I get it. So you're trying to "pre-inflate" the whole cluster of > shards and entities, and they're expected to be essentially static after > that? Interesting problem. > > So basically, I'd recommend the previous advice to start with: try it out > the easy way, and see whether it works acceptably. It's potentially > straining the system, but it's not *obviously* a crisis, and trying it > shouldn't be too hard. > > > If that doesn't work, I *suspect* you can do this in a much lower-traffic > way, but it would take a *bunch* more hacking of the sharding system. > Consider: > > -- You have a lot of control over where a given message gets routed, > through the ExtractShardId function. That implicitly determines what > Shards exist: the possible Shards are exactly the possible return values of > that function. > -- You *can* take control over where a given Shard lives, by defining a > ShardAllocationStrategy. Folks usually just use the default, allowing > things to happen dynamically, but if you want more static control, you can > likely make that bend to your will. > > So, in order to inflate huge numbers of entities with minimal network > traffic, it seems like you could pre-decide on how you *want* the > allocation to work, build ExtractShardId and ShardAllocationStrategy > accordingly, and send "init" messages to each Entity separately from the > node where you are coercing it to live. That is, each node would be > responsible for "inflating" the Entities that are going to live on it > according to the ShardAllocationStrategy. > > I don't think I'd recommend this approach unless the situation is dire > enough to warrant it -- it's a lot of work, and requires you to take on a > lot of responsibility for the sharding -- but consider it food for thought > if things look problematic... > > On Fri, Jun 10, 2016 at 2:43 PM, kraythe <[email protected] <javascript:>> > wrote: > >> The problem is I need to have the actors running and from what I >> understand the sharding system doesn't start an actor until you actually >> pass a message to an actor with that id. They cant subscribe to the topic >> until they are started. Now I could have some mediator actor to start them >> but then I would have to know if the ID of the actor would end up on the >> current system. Otherwise I would be making a remote call anyway. >> >> On Wednesday, June 8, 2016 at 3:36:07 PM UTC-5, Rob Crawford wrote: >>> >>> As I recall, the distributed pub/sub sends a single message to each >>> node, then the message is distributed to the subscribers on those nodes. So >>> it's likely more efficient to use this method to ask the actors for their >>> state. More efficient both ways -- no million-iteration loop to generate >>> the messages, and no transmission of millions of messages. >>> >>> >>> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
