Hi Chelios, The problem you are solving is divided in two and I think it has been resolved before though it is quite complex but if you divide and conquer it might turn out to be easy. IMHO here are the main aspects of your problem:
- Your data is distributed, each node with data will return the result to the node querying it. - A query coordinator actor (one of these has to live on each node for the sake of saving round-trips) will send such query to each node and expect a list of "map reduced" result. The key is to "map reduce", I'm assuming you first want to get the list of actors that comply with your search criteria and then once you have such list do something with them or via "them" In that case you want a map reduce in-memory data structure per node holding data, assuming each node as a list of workers to parallel-ize the query the rest is simple: Some ideas in the following link: http://www.infoq.com/articles/ApacheCrunch HTH, Guido. On Wednesday, March 30, 2016 at 10:09:04 AM UTC+1, Chelios wrote: > > Hi Konrad, > > Your reply gave me the confidence to continue with implementing the Actor > based search. Thank You :D ... I'm doing this just for research purposes, > I'm just trying to see if I can get a high performant, distributed, > in-memory system by just using Eventsourcing with Akka Actors without using > any other external database or tool, other than an Eventstore database. > > Can I also attend the workshop ? Seeing something Actor design patterns to > designing a search engine architecture is something I need to learn for > this :) > > Cheers, > Chel > > On Wednesday, March 30, 2016 at 7:29:33 PM UTC+11, Konrad Malawski wrote: >> >> Technically it's doable , but I'm not sure if that'll reduce complexity >> :-) >> Search really has to be "good" in order to be useful, just "fast but bad >> results" often won't satisfy anyone, >> thus I'm not sure implementing your own custom search engine is a good >> idea (unless that is exactly the goal of your business >> – be a search engine). >> >> A fun fact, one of the workshops I do is basically that, a multi-tier >> search engine architecture, however it depends if your entire job is to >> build the search engine, or you just should use an out of the box one >> because it's one of the 100 things you work on :-) >> >> -- >> Cheers, >> Konrad 'ktoso’ Malawski >> <http://akka.io>Akka <http://akka.io> @ Lightbend <http://typesafe.com> >> <http://lightbend.com> >> >> On 30 March 2016 at 08:42:57, Chelios ([email protected]) wrote: >> >> Hey guys >> >> I've got an Eventsource based application (Not CQRS - Read and write are >> both on the write side). The state of all the entities/aggregates/actor are >> stored in memory because the data is not going to go above 120GB and I've >> have a machine with 265GB RAM. >> >> *Problem:* >> Suppose I have a million Products where each *Product* is an Actor >> supervised by *ProductSupervisorActor* and I want to perform the >> following query: >> *Query*: Find all the products where the *product description* matches >> some user input. >> >> I'm wondering if I could get away with just querying the state of the >> million actors and aggregating the result into one >> *SearchRequestHandlerActor* instead of using a search database like SOLR >> ? I've used SOLR before and it's super fast but I'm just trying to reduce >> the complexity in my application. If the state is already in memory may be >> I can just find a way to query it instead of introducing another moving >> part (SOLR) into the system that I have to manage and make sure that the >> data is synchronized. >> >> I would really like to find a solution to perform the above query >> efficiently by just using Actors with paging. If I can achieve this then I >> can have *ProductActor*s running anywhere in a cluster and the search >> would work just fine. Instead, if I was using SOLR I would have to shard or >> partition the database which just another hassle. >> >> RIght now I've got a *ProductSearchRequestHandlerActor* which*,* on >> initilization, accepts *totalNumberOfMessagesExpect: Long* and accepts >> messages of type *Option[ProductState]* until the >> *totalNumberOfMessagesExpect* is reached. *I have not implemented paging >> yet.* >> >> I just wanted to get your opinion or ideas on how I can achieve this >> efficiently or any tips or I'm being silly for trying this because there is >> no central index ? >> >> Chel >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> >> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
