Re: Injest pauses

2012-08-29 Thread Alexey Serba
Hey Brad, This leads me to believe that a single merge thread is blocking indexing from occuring. When this happens our producers, which distribute their updates amongst all the shards, pile up on this shard and wait. Which version of Solr you are using? Have you tried 4.0 beta? *

RE: Injest pauses

2012-08-29 Thread Voth, Brad (GE Corporate)
[ase...@gmail.com] Sent: Wednesday, August 29, 2012 6:45 AM To: solr-user@lucene.apache.org Subject: Re: Injest pauses Hey Brad, This leads me to believe that a single merge thread is blocking indexing from occuring. When this happens our producers, which distribute their updates amongst all

RE: Injest pauses

2012-08-29 Thread Voth, Brad (GE Corporate)
. -Original Message- From: Voth, Brad (GE Corporate) Sent: Wednesday, August 29, 2012 10:17 AM To: solr-user@lucene.apache.org Subject: RE: Injest pauses Very interesting links, after much more digging yesterday this appears to be exactly what I'm seeing. I am using 4.0 beta currently for my

Re: Injest pauses

2012-08-29 Thread Yonik Seeley
On Wed, Aug 29, 2012 at 11:58 AM, Voth, Brad (GE Corporate) brad.v...@ge.com wrote: Anyone know the actual status of SOLR-2565, it looks to be marked as resolved in 4.* but I am still seeing long pauses during commits using 4.* SOLR-2565 is definitely committed - adds are no longer blocked by

RE: Injest pauses

2012-08-29 Thread Voth, Brad (GE Corporate)
Thanks, I'll continue with my testing and tracking down the block. -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Wednesday, August 29, 2012 12:28 PM To: solr-user@lucene.apache.org Subject: Re: Injest pauses On Wed, Aug 29, 2012

RE: Injest pauses

2012-08-29 Thread Voth, Brad (GE Corporate)
To: solr-user@lucene.apache.org Subject: RE: Injest pauses Thanks, I'll continue with my testing and tracking down the block. -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Wednesday, August 29, 2012 12:28 PM To: solr-user@lucene.apache.org

Re: Injest pauses

2012-08-29 Thread Alexey Serba
: Voth, Brad (GE Corporate) Sent: Wednesday, August 29, 2012 12:32 PM To: solr-user@lucene.apache.org Subject: RE: Injest pauses Thanks, I'll continue with my testing and tracking down the block. -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik

Re: Injest pauses

2012-08-29 Thread Otis Gospodnetic
@lucene.apache.org solr-user@lucene.apache.org Cc: Sent: Wednesday, August 29, 2012 1:05 PM Subject: RE: Injest pauses Interestingly it is not pausing during every commit so at least a portion of the time the async commit code is working.  Trying to track down the case where a wait would still be issued

RE: Injest pauses

2012-08-28 Thread Voth, Brad (GE Corporate)
, August 27, 2012 4:54 PM To: solr-user@lucene.apache.org Subject: Injest pauses Hello all, I'm working on implementing a solution in a very high index rate, lower query rate project. We have a very small record size, 2 small strings, 6 longs, 7 ints, and 2 dates, indexing on 8 of those fields. We

Re: Injest pauses

2012-08-28 Thread Shawn Heisey
On 8/27/2012 2:54 PM, Voth, Brad (GE Corporate) wrote: Hello all, I'm working on implementing a solution in a very high index rate, lower query rate project. We have a very small record size, 2 small strings, 6 longs, 7 ints, and 2 dates, indexing on 8 of those fields. We need to be able to

RE: Injest pauses

2012-08-28 Thread Voth, Brad (GE Corporate)
if the flushing is able to keep up at some point. From: Shawn Heisey [s...@elyograg.org] Sent: Tuesday, August 28, 2012 1:37 PM To: solr-user@lucene.apache.org Subject: Re: Injest pauses On 8/27/2012 2:54 PM, Voth, Brad (GE Corporate) wrote: Hello all, I'm working

Injest pauses

2012-08-27 Thread Voth, Brad (GE Corporate)
Hello all, I'm working on implementing a solution in a very high index rate, lower query rate project. We have a very small record size, 2 small strings, 6 longs, 7 ints, and 2 dates, indexing on 8 of those fields. We need to be able to maintain an injest rate of 50k records/sec average.