Thanks Eric. For the record, we are using 1.4.1 and SolrJ.
On 31 October 2010 01:54, Erick Erickson erickerick...@gmail.com wrote:
What version of Solr are you using?
About committing. I'd just let the solr defaults handle that. You configure
this in the autocommit section of solrconfig.xml. I'm pretty sure this
gets
triggered even if you're using SolrJ.
That said, it's probably wise to issue a commit after all your data is
indexed
too, just to flush any remaining documents since the last autocommit.
Optimize should not be issued until you're all done, if at all. If
you're not deleting (or updating) documents, don't bother to optimize
unless the number of files in your index directory gets really large.
Recent Solr code almost removes the need to optimize unless you
delete documents, but I confess I don't know the revision number
recent refers to, perhaps only trunk...
HTH
Erick
On Thu, Oct 28, 2010 at 9:56 AM, Savvas-Andreas Moysidis
savvas.andreas.moysi...@googlemail.com wrote:
Hello,
We currently index our data through a SQL-DIH setup but due to our model
(and therefore sql query) becoming complex we need to index our data
programmatically. As we didn't have to deal with commit/optimise before,
we
are now wondering whether there is an optimal approach to that. Is there
a
batch size after which we should fire a commit or should we execute a
commit
after indexing all of our data? What about optimise?
Our document corpus is 4m documents and through DIH the resulting index
is
around 1.5G
We have searched previous posts but couldn't find a definite answer. Any
input much appreciated!
Regards,
-- Savvas
