On Mon, May 18, 2009 at 8:18 AM, Otis Gospodnetic <otis_gospodne...@yahoo.com> wrote: > > Chris, > > As far as I know, AOL is using Solr with lots of cores. What I don't know is > how they are handling shutting down of idle cores, which is something you'll > need to do if your machine can't handle all cores being open and their data > structures being populated at all times. I know I had to do that same for > Simpy. :)
we have a custom build of Solr. we do just in time automatic loading of cores and an LRU based unloading of cores when the upper water mark is crossed > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > ----- Original Message ---- >> From: Chris Cornell <srchn...@gmail.com> >> To: solr-user@lucene.apache.org >> Sent: Sunday, May 17, 2009 10:11:10 PM >> Subject: Re: multicore for 20k users? >> >> On Sun, May 17, 2009 at 8:38 PM, Otis Gospodnetic >> wrote: >> > >> > Chris, >> > >> > Yes, disk space is cheap, and with so little overlap you won't gain much by >> putting everything in a single index. Plus, when each user has a separate >> index, it's easy to to split users and distribute over multiple machines if >> you >> ever need to do that, it's easy and fast to completely reindex one user's >> data >> without affecting other users, etc. >> > >> > Several years ago I built Simpy at http://www.simpy.com/ that way (but >> pre-Solr, so it uses Lucene directly) and never regretted it. There are way >> more than 20K users there with many searches per second and with constant >> indexing. Each user has an index for bookmarks and an index for notes. Each >> group has its own index, shared by all group members. The main bookmark >> search >> is another index. People search is yet another index. And so on. Single >> server. >> > >> >> Thankyou very much for your insight and experience, sounds like we >> shouldn't be thinking about "prematurely optimizing" this. >> >> Has someone actually used multicore this way, though? With thousands of >> them? >> >> Independently of advice in that regard, I guess our next step is to >> explore and create some "dummy" scenarios/tests to try and stress >> multicore (search latency is not as much of a factor as memory usage >> is). I'll report back on any conclusion we come to. >> >> Thanks! >> Chris > > -- ----------------------------------------------------- Noble Paul | Principal Engineer| AOL | http://aol.com
