Roland, thanks for the response, been a few months but I am back on this problem.
So what I have done now is split the filters, map[string, string] over my filter actors. The filter function will return a 1 or 0 and I fold over the responses and if its > 0 I filter the message. Right now I am just maintaining the list of refs and "asking" them all. I would love to do this with a router as it would make everything a bit cleaner. Ideally I would add the filters in a round robin fashion and send a broadcast of the messages I would like to filter. However, obviously I want to wait for all responses. Is there a way to do this out of the box with routers? Thanks, Bobby On Tuesday, June 3, 2014 12:38:26 AM UTC-5, rkuhn wrote: > > Hi Bobby, > > the details are use-case specific, but here are a few guidelines: > > > - filter can be parallelized if the order of messages after the filter > does not matter; if it does then the merging step will eat up some of the > parallelization gains > - parallelization can easily be done using a Router > - filtering is CPU-bound, so you should not parallelize beyond #cores > you want to dedicate to this > - you are creating a lot of intermediate collections; cutting down on > the garbage generation will help a lot (i.e. use .iterator, .exists and > .forall) > - your setOfFilters can be pruned by removing all maps whose keySet is > a superset of any other maps’s keySet > > > If you need more assistance with your specific solution we (Typesafe) also > offer commercial help. > > Regards, > > Roland > > 30 maj 2014 kl. 21:06 skrev Bobby Richards <[email protected] > <javascript:>>: > > Wanted to get some feedback on this problem.... > > I have json messages coming in at a rate of about 1000/second which are > then converted to map[String, String]; anywhere from 10-50 keys. I want to > filter those messages based on an unknown number of filters. Filters will > be any combination of those keys also in the form of map[string, string]. > Technically the number of filters could be limitless but lets say real > world max is 5000. > > I was thinking of something like this for my filter function which in a > brute force manner just iterates through the set of filters and applies to > each incoming message. > > var setOfFilters: Set[Map[String, String]] = Set.empty > > def receive = { > case message(msg) => filter(msg) //send on or do something > case AddFilter(f) => // > case RemoveFilter(f) => // > > def filter(common: Map[String, String]): Boolean = { > def inner(f: Set[Map[String, String]]): Boolean = { > f.headOption match { > case Some(head) => > if(head == common.filterKeys(head.keySet)) true > else if(f.tail.size > 0) inner(f.tail) > else false > case None => false > } > } > > inner(setOfFilters) > } > > clearly this is not optimal. I have thought of multiple options like > finding the most common field from the filters and checking that first, or > trying to logically group the filters from top down and create actors for > those with sub actor etc. But I would really like for it to be completely > dynamic, meaning I am not providing any "help" like if(common("company" == > "someco") msg forward someRef. > > Another thought was to just create one actor per filter and place behind a > broadcast router, or perhaps consolidate to a number of filters (10?) per > actor with acceptable performance behind a broadcast router as well. > filters are simple pass/fail and if there is no filter message is passed > on to some other ref. I would have to collect the results though I am not > sure there is a way around that. > > Any help or thoughts are greatly appreciated. > > Bobby > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > > > > *Dr. Roland Kuhn* > *Akka Tech Lead* > Typesafe <http://typesafe.com/> – Reactive apps on the JVM. > twitter: @rolandkuhn > <http://twitter.com/#!/rolandkuhn> > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
