Solr DIH - Small index still take time?

2013-05-23 Thread Spadez
Hi,

This is the situation, I have two sources of data in my dataimport handler,
one is huge, the other is tiny:

Source A: 10-20 records
Source B: 50,000,000 records

I was wondering what happens if I was to do a DIH just on Source A every 10
mins, and only run the DIH on source B every 24 hours.

Would running my DIH on Source A be extremely quick, because the data we are
importing is small, or would it still be time consuming, because it would
have to rebuild the index of the entire SOLR (i.e 50,000,010 records).

Thank you!



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-DIH-Small-index-still-take-time-tp4065582.html
Sent from the Solr - User mailing list archive at Nabble.com.


Re: Solr DIH - Small index still take time?

2013-05-23 Thread Alexandre Rafalovitch
That should work. Just watch out for (set value of)
preImportDeleteQuery. Otherwise, when you do full import you may
accidentally delete items from the other set.

Regards,
   Alex.
Personal blog: http://blog.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all
at once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
book)


On Thu, May 23, 2013 at 6:25 AM, Spadez james_will...@hotmail.com wrote:
 Hi,

 This is the situation, I have two sources of data in my dataimport handler,
 one is huge, the other is tiny:

 Source A: 10-20 records
 Source B: 50,000,000 records

 I was wondering what happens if I was to do a DIH just on Source A every 10
 mins, and only run the DIH on source B every 24 hours.

 Would running my DIH on Source A be extremely quick, because the data we are
 importing is small, or would it still be time consuming, because it would
 have to rebuild the index of the entire SOLR (i.e 50,000,010 records).

 Thank you!



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Solr-DIH-Small-index-still-take-time-tp4065582.html
 Sent from the Solr - User mailing list archive at Nabble.com.