One of the biggest bangs you can get out of indexing is to multi-
thread it pretty heavily. Solr can accept lots of simultaneous
connections.
Folks worried about the HTTP communication are typically those that
are new to Solr and see that as a bottleneck without measuring.
Those that have done the measuring have found that HTTP is really a
huge factor on indexing performance.
When Lucene 2.3 is dropped into Solr (coming soon), indexing speed
will improve even more substantially thanks to core improvements at
the heart.
But, POSTing more than one document at a time also good advice.
Erik
On Nov 23, 2007, at 7:03 AM, Ewout Van Troostenberghe wrote:
How do you fill the index? Our main database has about 700,000
records
and I don't know if I should build one huge XML-file and feed
that into
SOLR or use a script that sends one record at a time with a
commit after
every 1000 records or so. Or do something in between and split it
into
chunks of a few thousand records each? What are your experiences?
What
if a record gives an error? Will the whole file be recjected or just
that one record?
There is a Java command line tool or you can see the VuFind's
solution. If you can, I suggest you to prefer a pure java
solution, writing directly to the Solr index (with the Solr
API), because its much-much more quicker than the PHP's
(Rail's, Perl's) solution which based on a web-service
(which need the PHP parsing and HTTP request curve).
The PHP solution does nothing with Solr directly, it
use the web service, and all the code can be rewriten
in Perl.
When you want use a scripting language to fill the solr index, rather
then using solr API directly, you should consider buffering as an
intermediate solution. It can speed up indexing with orders of
magnitude. Create your XML in the script, and keep them in-memory
until
you have 50 or 100 documents. Then post these together.
Attached is a small ruby script we use to do solr indexing. It reads
yaml records from standard input, does some processing (burried in our
libraries), buffers the result and posts after 100 records are
gathered.
Regards,
Ewout
<index_solr_fast.rb>
