It's a 3GHz Intel box with Xeon processors, 64GB ram :) Peter
On 1/25/06, Yonik Seeley <[EMAIL PROTECTED]> wrote: > > Thanks Peter, that's useful info. > > Just out of curiosity, what kind of box is this? what CPUs? > > -Yonik > > On 1/25/06, Peter Keegan <[EMAIL PROTECTED]> wrote: > > This is just fyi - in my stress tests on a 8-cpu box (that's 8 real > cpus), > > the maximum throughput occurred with just 4 query threads. The query > > throughput decreased with fewer than 4 or greater than 4 query threads. > The > > entire index was most likely in the file system cache, too. Periodic > > snapshots of stack traces showed most threads blocked in the > synchronization > > in: FSIndexInput.readInternal(), when the thread count exceeded 4. > > > > Peter > > > > > > On 11/22/05, Oren Shir <[EMAIL PROTECTED]> wrote: > > > > > > Hi, > > > > > > There are two sunchronization points: on the stream and on the reader. > > > Using > > > different FSDirectoriy and IndexReaders should solve this. I'll let > you > > > know > > > once I code it. Right now I'm checking if making my Documents store > less > > > data will move the bottleneck to some other place. > > > > > > Thanks again, > > > Oren Shir > > > > > > On 11/21/05, Doug Cutting <[EMAIL PROTECTED]> wrote: > > > > > > > > Jay Booth wrote: > > > > > I had a similar problem with threading, the problem turned out to > be > > > > that in > > > > > the back end of the FSDirectory class I believe it was, there was > a > > > > > synchronized block on the actual RandomAccessFile resource when > > > reading > > > > a > > > > > block of data from it... high-concurrency situations caused > threads to > > > > stack > > > > > up in front of this synchronized block and our CPU time wound up > being > > > > spent > > > > > thrashing between blocked threads instead of doing anything > useful. > > > > > > > > This is correct. In Lucene, multiple streams per file are created by > > > > cloning, and all clones of an FSDirectory input stream share a > > > > RandomAccessFile and must synchronize input from it. MmapDirectory > does > > > > not have this limitation. If your indexes are less than a few GB or > you > > > > are using 64-bit hardware, then MmapDirectory should work well for > you. > > > > Otherwise it would be simple to write an nio-based Directory that > does > > > > not use mmap that is also unsynchronized. Such a contribution would > be > > > > welcome. > > > > > > > > > Making multiple IndexSearchers and FSDirectories didn't help > because > > > in > > > > the > > > > > back end, lucene consults a singleton HashMap of some kind (don't > > > > remember > > > > > implementation) that maintained a single FSDirectory for any given > > > index > > > > > being accessed from the JVM... multiple calls to > > > > FSDirectory.getDirectory > > > > > actually return the same FSDirectory object with synchronization > at > > > the > > > > same > > > > > point. > > > > > > > > This does not make sense to me. FSDirectory does keep a cache of > > > > FSDirectory instances, but i/o should not be synchronized on these. > One > > > > should be able to open multiple input streams on the same file from > an > > > > FSDirectory. But this would not be a great solution, since file > handle > > > > limits would soon become a problem. > > > > > > > > Doug > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > >
