Hi Bobby, the details are use-case specific, but here are a few guidelines:
filter can be parallelized if the order of messages after the filter does not matter; if it does then the merging step will eat up some of the parallelization gains parallelization can easily be done using a Router filtering is CPU-bound, so you should not parallelize beyond #cores you want to dedicate to this you are creating a lot of intermediate collections; cutting down on the garbage generation will help a lot (i.e. use .iterator, .exists and .forall) your setOfFilters can be pruned by removing all maps whose keySet is a superset of any other maps’s keySet If you need more assistance with your specific solution we (Typesafe) also offer commercial help. Regards, Roland 30 maj 2014 kl. 21:06 skrev Bobby Richards <[email protected]>: > Wanted to get some feedback on this problem.... > > I have json messages coming in at a rate of about 1000/second which are then > converted to map[String, String]; anywhere from 10-50 keys. I want to filter > those messages based on an unknown number of filters. Filters will be any > combination of those keys also in the form of map[string, string]. > Technically the number of filters could be limitless but lets say real world > max is 5000. > > I was thinking of something like this for my filter function which in a brute > force manner just iterates through the set of filters and applies to each > incoming message. > > var setOfFilters: Set[Map[String, String]] = Set.empty > > def receive = { > case message(msg) => filter(msg) //send on or do something > case AddFilter(f) => // > case RemoveFilter(f) => // > > def filter(common: Map[String, String]): Boolean = { > def inner(f: Set[Map[String, String]]): Boolean = { > f.headOption match { > case Some(head) => > if(head == > common.filterKeys(head.keySet)) true > else if(f.tail.size > 0) inner(f.tail) > else false > case None => false > } > } > > inner(setOfFilters) > } > > clearly this is not optimal. I have thought of multiple options like finding > the most common field from the filters and checking that first, or trying to > logically group the filters from top down and create actors for those with > sub actor etc. But I would really like for it to be completely dynamic, > meaning I am not providing any "help" like if(common("company" == "someco") > msg forward someRef. > > Another thought was to just create one actor per filter and place behind a > broadcast router, or perhaps consolidate to a number of filters (10?) per > actor with acceptable performance behind a broadcast router as well. filters > are simple pass/fail and if there is no filter message is passed on to some > other ref. I would have to collect the results though I am not sure there is > a way around that. > > Any help or thoughts are greatly appreciated. > > Bobby > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. Dr. Roland Kuhn Akka Tech Lead Typesafe – Reactive apps on the JVM. twitter: @rolandkuhn -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
