If you can make that solution work for you, I think it is a wise one which will serve you well. In some cases that solution won't work, because you _need_ the frequently changing data in Solr to be searched against in Solr. But if you can get away without that, I think you will be well-served by keeping any data that doesn't need to be searched against by Solr in an external non-Solr store. It's really rarely a bad plan to just put in Solr what needs to be searched against in Solr -- whether or not the 'other' stuff changes frequently.
Only you (if anyone!) know enough about your requirements and plans to know how much of a problem it will be to have your 'mutable' data not in Solr, and thus not searchable with Solr. ________________________________________ From: Bing Li [lbl...@gmail.com] Sent: Friday, March 04, 2011 3:21 PM To: Michael McCandless Cc: solr-user@lucene.apache.org Subject: Re: When Index is Updated Frequently Dear Michael, Thanks so much for your answer! I have a question. If Lucene is good at updating, it must more loads on the Solr cluster. So in my system, I will leave the large amount of crawled data unchanged for ever. Meanwhile, I use a traditional database to keep mutable data. Fortunately, in most Internet systems, the amount of mutable data is much less than that of immutable one. How do you think about my solution? Best, LB On Sat, Mar 5, 2011 at 2:45 AM, Michael McCandless < luc...@mikemccandless.com> wrote: > On Fri, Mar 4, 2011 at 10:09 AM, Bing Li <lbl...@gmail.com> wrote: > > > According to my experiences, when the Lucene index updated frequently, > its > > performance must become low. Is it correct? > > In fact Lucene can gracefully handle a high rate of updates with low > latency turnaround on the readers, using the near-real-time (NRT) API > -- IndexWriter.getReader() (or in soon-to-be 31, > IndexReader.open(IndexWriter)). > > NRT is really something a hybrid of "eventual consistency" and > "immediate consistency", because it lets your app have full control > over how quickly changes must be visible by controlling when you > pull a new NRT reader. > > That said, Lucene can't offer true immediate consistency at a high > update rate -- the time to open a new NRT reader is usually too costly > to do, eg, for every search. But eg every 100 msec (say) is > reasonable (depending on many variables...). > > So... for your app you should run some tests and see. And please > report back. > > (But, unfortunately, NRT hasn't been exposed in Solr yet...). > > -- > Mike > > http://blog.mikemccandless.com >
