Kick off some indexing more than once - eg, post a folder of docs, and
while thats working, post another.
I've been thinking about a multi threaded UpdateProcessor as well - that
could be interesting.
- Mark
sunnyfr wrote:
Hi,
I was reading this post and I wondering how can I parallelize document
processing???
Thanks Erik
Erik Hatcher wrote:
On Feb 21, 2007, at 4:25 PM, Jack L wrote:
couple of times today at around 158 documents / sec.
This is not bad at all. How about search performance?
How many concurrent queries have people been having?
What does the response time look like?
I'm the only user :) What I've done is a proof-of-concept for our
library. We have 3.7M records that I've indexed and faceted. Search
performance (in my unrealistic single user scenario) is blazing (50ms
or so) for purely full-text queries. For queries that return facets,
the response times are actually quite good too (~900ms, or less
depending on the request) - provided the filter cache is warmed and
large enough. This is running on my laptop (MacBook Pro, 2GB RAM,
1.83GHz) - I'm sure on a beefier box it'll only get better.
Thanks to the others that clarified. I run my indexers in
parallel... but a single instance of Solr (which in turn handles
requests in parallel as well).
Do you feel if multi-threaded posting is helpful?
It depends. If the data processing can be parallelized and your
hardware supports it, it can certainly make a big difference... it
did in my case. Both CPUs were cooking during my parallel indexing
runs.
Erik
