Re: Parallel indexing in Solr

2012-02-07 Thread Per Steffensen
You could try to isolate the bottleneck by testing the indexing speed from the local machine hosting Solr. Also tools like iostat or sar might give you more details about the disk side. Yes, I am doing different stuff to isolate bottleneck. Im also profiling JVM. And I am using iostat, top a

Re: Parallel indexing in Solr

2012-02-07 Thread Sami Siren
On Mon, Feb 6, 2012 at 5:55 PM, Per Steffensen wrote: > Sami Siren skrev: > >> On Mon, Feb 6, 2012 at 2:53 PM, Per Steffensen >> wrote: >> >> >> >>> >>> Actually right now, I am trying to find our what my bottleneck is. The >>> setup >>> is more complex, than I would bother you with, but basicall

Re: Parallel indexing in Solr

2012-02-06 Thread Erick Erickson
. I've had recurring discussions with "executive level folks" that no matter how many VMs you host on a machine, and no matter how big that machine is, there really, truly, *is* some hardware underlying it all that really, truly, *does* have some limits. And adding more VMs doesn't somehow get aro

Re: Parallel indexing in Solr

2012-02-06 Thread Per Steffensen
Sami Siren skrev: On Mon, Feb 6, 2012 at 2:53 PM, Per Steffensen wrote: Actually right now, I am trying to find our what my bottleneck is. The setup is more complex, than I would bother you with, but basically I have servers with 80-90% IO-wait and only 5-10% "real CPU usage". It might not

Re: Parallel indexing in Solr

2012-02-06 Thread Per Steffensen
So SolrJ with CommonsHttpSolrServer will not support handling several requests concurrently? Nope. Use StreamingUpdateSolrServer, it should be just a drop-in with a different constructor. I will try to do that. It is a little bit difficult for me, as we are actually not dealing with

Re: Parallel indexing in Solr

2012-02-06 Thread Sami Siren
On Mon, Feb 6, 2012 at 2:53 PM, Per Steffensen wrote: > Actually right now, I am trying to find our what my bottleneck is. The setup > is more complex, than I would bother you with, but basically I have servers > with 80-90% IO-wait and only 5-10% "real CPU usage". It might not be a > Solr-relat

Re: Parallel indexing in Solr

2012-02-06 Thread Erick Erickson
Right. See below. On Mon, Feb 6, 2012 at 7:53 AM, Per Steffensen wrote: > See response below > > Erick Erickson skrev: > >> Unfortunately, the answer is "it depends(tm)". >> >> First question: How are you indexing things? SolrJ? post.jar? >> > > SolrJ, CommonsHttpSolrServer > >> But some observat

Re: Parallel indexing in Solr

2012-02-06 Thread Per Steffensen
See response below Erick Erickson skrev: Unfortunately, the answer is "it depends(tm)". First question: How are you indexing things? SolrJ? post.jar? SolrJ, CommonsHttpSolrServer But some observations: 1> sure, using multiple cores will have some parallelism. So will using a single co

Re: Parallel indexing in Solr

2012-02-03 Thread Erick Erickson
Unfortunately, the answer is "it depends(tm)". First question: How are you indexing things? SolrJ? post.jar? But some observations: 1> sure, using multiple cores will have some parallelism. So will using a single core but using something like SolrJ and StreamingUpdateSolrServer. Especial

Parallel indexing in Solr

2012-02-03 Thread Per Steffensen
Hi This topic has probably been covered before, but I havnt had the luck to find the answer. We are running solr instances with several cores inside. Solr running out-of-the-box on top of jetty. I believe jetty is receiving all the http-requests about indexing ned documents, and forwards it