See here: http://doc.akka.io/docs/akka/2.3.9/contrib/cluster-sharding.html
You should also look at persistence: http://doc.akka.io/docs/akka/2.3.9/scala/persistence.html -Endre On Tue, Apr 21, 2015 at 4:21 PM, <[email protected]> wrote: > How cool stuff is this? never heard of Akka Sharding before. Seems to fit > exactly my usecase. Seems to be my lucky day > > Am Dienstag, 21. April 2015 15:46:43 UTC+2 schrieb drewhk: > >> >> >> On Tue, Apr 21, 2015 at 3:22 PM, <[email protected]> wrote: >> >>> >>> thanks. I would like to ensure that when new Nodes are added they do not >>> take their share of objects from all the other cache machines. >>> I guess I would need to implement my own caching algo for this to work. >>> >> >> How would that work? Assuming that you have 3 nodes, and you already >> decided which objects map to which nodes, then you have a mapping M1 from >> all objects to the nodes {N1, N2 N3}. Now if you add a new node, then that >> mapping needs to be changed since it was *total*: >> - all nodes have objects that are mapped to them (at least ideally) >> - all objects map to exactly one node >> >> Now you will need to maintain a new mapping M2 to {N1, N2, N3, N4} *in >> addtion*, *and* the difference between M1(object) and M2(object). The >> reason is: >> - You need mapping M1 to be able to access object that were mapped to >> N1-3 before the node N4 has been added >> - You need mapping M2 to know where to put new objects >> - You need to know if certain object have been added already before N4 >> came online, or later (since in the former case you need M1 to find its >> location, and in the latter case you need M2 to find its location) >> >> And this gets worse with any new additon and removal. There is a reason >> why consistent hashing has been invented. Btw, this is a problem that >> cluster sharding solves, but it also needs to migrate things, and the only >> way you can make that happen is using Akka persistence with it. >> >> Akka Sharding uses the problem outlined above in the following way: >> - It maintains a location map, but not for individual entries, but >> buckets (called shards) >> - this map is maintained resiliently in a distributed way (it is a >> singleton that can migrate when the host node fails, and clients can find >> it again at the new place) >> - the hash function maps objects to buckets, i.e. there is a simple >> mapping from object to bucket by using a hash function, and there is an >> index that is actively maintained that maps buckets to nodes. The number of >> buckets (shards) are always fixed, but the number of nodes can change. >> - the buckets are distributed among available nodes >> - when nodes are added, some buckets are moved to those nodes, and the >> singleton updates the index, so all nodes know where the bucket is located >> - when nodes are removed or fail, all the buckets hosted by those nodes >> will be distributed and restored from persistent storage at the remaining >> nodes. >> >> -Endre >> >> >>> >>> >>> Am Dienstag, 21. April 2015 12:09:37 UTC+2 schrieb drewhk: >>>> >>>> Hi John, >>>> >>>> On Tue, Apr 21, 2015 at 11:14 AM, <[email protected]> wrote: >>>> >>>>> akka.actor.deployment { >>>>> /myRouter { >>>>> router = consistent-hashing-group >>>>> nr-of-instances = 10000 >>>>> >>>>> routees.paths = ["/user/worker"] >>>>> cluster { >>>>> max-nr-of-instances-per-node = 1 >>>>> enabled = on >>>>> allow-local-routees = off >>>>> use-role = recipient >>>>> } >>>>> } >>>>> } >>>>> >>>>> If I start with for example 3 worker nodes "A" "B" "C" the >>>>> consistent-hashing-group algorithm works as expected: >>>>> If a worker on node "B" recievies a message with a consistentHashKey = >>>>> "123" this message is always send to "B" >>>>> >>>>> But if I add a Node D from now on the message consistentHashKey = "123" >>>>> is send to D. >>>>> >>>>> Is this the expected behaviour? >>>>> >>>>> Yes it is, this is how consistent hashing works: >>>> https://en.wikipedia.org/wiki/Consistent_hashing >>>> >>>> Quoting: >>>> "Consistent hashing maps objects to the same cache machine, as far as >>>> possible. It means when a cache machine is added, it takes its share of >>>> objects from all the other cache machines and when it is removed, its >>>> objects are shared between the remaining machines." >>>> >>>> The goal of consistent caching is to: >>>> - not to store a global index of all key->node mappings (that would be >>>> needed in your example not to remap "123" to another node) because it is >>>> very expensive and fragile >>>> - remap only part of the keyspace between nodes, most objects are >>>> mapped to the node they were mapped before when a node is added or removed. >>>> >>>> -Endre >>>> >>>> >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: >>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>> >>>>>>>>>> Search the archives: >>>>> https://groups.google.com/group/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: >>> http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
