Hi Steve, On Sun, Feb 8, 2015 at 8:46 PM, Steve Ramage <[email protected]> wrote:
> Hello, > > I haven't fully digested the documentation yet, so please forgive me if > this question is ill-posed. I have a basic many producers many consumers > problem, where producers and consumers may exist on the same node or may > not. Additionally producers may produce a number of tasks simultaneously, > or none at other times. The producers exist isolated from one another > entirely, and are best viewed as different applications levering efficiency > gains than being part of one large application that has invariants to > maintain, although all of these applications would have a single owner and > all of them can be considered trust worthy. Both the producers and > consumers will be scheduled on a time sharing cluster, so you may have 100 > jobs that trickle in over time, each one having a time limit and shutting > down when it's time limit is reached, so when it starts up only say the > first 25 jobs are up, then another 40 may start, then another 10 may come > in, then the first 25 die, then the next 25 come in. What I would like to > do is have a way of honouring both FIFO but also support different > priorities for producers that gives the producer the ability to say that > certain jobs are higher priority. From my reading it seems that the best > system for this, is to use the cluster module and also the cluster > singleton pattern, which can handle assignments of producers to consumers. > Then once the assignment is done, the communication is direct between the > produce and consumer. When the consumer is done it messages the singleton > that it is free again. The producers keep all the tasks they have to do > outstanding, but tell the singleton some of the tasks (say the first 1000). > > Because these clusters are relatively short lived and very elastic, I'm > not particularly worried about network partitions or other weird failures, > as I think that would break the application in other ways. But if a > partition does happen, I noticed that the documentation says that multiple > singletons may be created, and that is actually exactly what I would want > to happen. > If you use auto-downing feature you can split the cluster in two halves, each one downing and removing the other half. Two separate clusters have been formed, each one hosting one singleton instance. > What I'm unclear about is if the partition is resolved what happens to the > singletons. > After downing there is no way back. The two separate clusters will not join each other again. You have to shutdown one cluster and startup/join the nodes to the running cluster. Note that members can be marked as unreachable and then they can become reachable again, but that is before the final decision of downing a member. > I guess what would be ideal is if both singletons died, and then another > was created. Since then I could just handle this case the same as if the > node with the singleton died. > > I guess another question I have, is that it says the singleton will always > migrate to the next oldest node. Is there a way to change this. I would > argue that in my use case 98% of the time, it would actually be best to > migrate to the youngest node, and that in some cases (say jobs trickle in 1 > per minute, and expire at the same rate), this would just be terrible. > That is not possible to change. Since you don't seem to have the strict requirement to run "at most one" singleton instance you are in a pretty good position to implement this yourself by subscribing to the cluster membership events. Note that the member addresses are ordered, and you can decide to run the master on the lowest/highest address, until it becomes unreachable and then you spawn another one, and so on. http://doc.akka.io/docs/akka/2.3.9/scala/cluster-usage.html#Subscribe_to_Cluster_Events Cheers, Patrik > > Thanks, > > Steve Ramage > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
