Thanks Otis. Could you write to same core (same index) from multiple threads at the same time? I thought each writer would lock the index so other can not write at the same time. I'll try it though.
Another reason of putting indexes in separate core was to limit the index size. Our index can grow up to 50G a day, so I was hoping writing to smaller indexes would be faster in separate cores and if needed I can merge them at later point (like end of day). I want to keep daily cores. Isn't this a good idea? How else can I limit the index size (beside multiple instances or separate boxes). Thanks, -vivek On Tue, Mar 31, 2009 at 8:28 PM, Otis Gospodnetic <otis_gospodne...@yahoo.com> wrote: > > Let me start with 4) > Have you tried simply using multiple threads to send your docs to a single > Solr instance/core? You should get about the same performance as what you > are trying with your approach below, but without the headache of managing > multiple cores and index merging (not yet possible to do programatically). > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > ----- Original Message ---- >> From: vivek sar <vivex...@gmail.com> >> To: solr-user@lucene.apache.org >> Sent: Tuesday, March 31, 2009 1:59:01 PM >> Subject: Merging Solr Indexes >> >> Hi, >> >> As part of speeding up the index process I'm thinking of spawning >> multiple threads which will write to different temporary SolrCores. >> Once the index process is done I want to merge all the indexes in >> temporary cores to a master core. For ex., if I want one SolrCore per >> day then every index cycle I'll spawn 4 threads which will index into >> some temporary index and once they are done I want to merge all these >> into the day core. My questions, >> >> 1) I want to use the same schema and solrconfig.xml for all cores >> without duplicating them - how do I do that? >> 2) How do I merge the temporary Solr cores into one master core >> programmatically? I've read the wiki on "MergingSolrIndexes", but I >> want to do it programmatically (like in Lucene - >> writer.addIndexes(..)) once the temporary indices are done. >> 3) Can I remove the temporary indices once the merge process is done? >> 4) Is this the right strategy to speed up indexing? >> >> Thanks, >> -vivek > >
