Wanted to get some feedback on this problem....
I have json messages coming in at a rate of about 1000/second which are
then converted to map[String, String]; anywhere from 10-50 keys. I want to
filter those messages based on an unknown number of filters. Filters will
be any combination of those keys also in the form of map[string, string].
Technically the number of filters could be limitless but lets say real
world max is 5000.
I was thinking of something like this for my filter function which in a
brute force manner just iterates through the set of filters and applies to
each incoming message.
var setOfFilters: Set[Map[String, String]] = Set.empty
def receive = {
case message(msg) => filter(msg) //send on or do something
case AddFilter(f) => //
case RemoveFilter(f) => //
def filter(common: Map[String, String]): Boolean = {
def inner(f: Set[Map[String, String]]): Boolean = {
f.headOption match {
case Some(head) =>
if(head == common.filterKeys(head.keySet)) true
else if(f.tail.size > 0) inner(f.tail)
else false
case None => false
}
}
inner(setOfFilters)
}
clearly this is not optimal. I have thought of multiple options like
finding the most common field from the filters and checking that first, or
trying to logically group the filters from top down and create actors for
those with sub actor etc. But I would really like for it to be completely
dynamic, meaning I am not providing any "help" like if(common("company" ==
"someco") msg forward someRef.
Another thought was to just create one actor per filter and place behind a
broadcast router, or perhaps consolidate to a number of filters (10?) per
actor with acceptable performance behind a broadcast router as well.
filters are simple pass/fail and if there is no filter message is passed
on to some other ref. I would have to collect the results though I am not
sure there is a way around that.
Any help or thoughts are greatly appreciated.
Bobby
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
