Thanks Peter, that's useful info. Just out of curiosity, what kind of box is this? what CPUs?
-Yonik On 1/25/06, Peter Keegan <[EMAIL PROTECTED]> wrote: > This is just fyi - in my stress tests on a 8-cpu box (that's 8 real cpus), > the maximum throughput occurred with just 4 query threads. The query > throughput decreased with fewer than 4 or greater than 4 query threads. The > entire index was most likely in the file system cache, too. Periodic > snapshots of stack traces showed most threads blocked in the synchronization > in: FSIndexInput.readInternal(), when the thread count exceeded 4. > > Peter > > > On 11/22/05, Oren Shir <[EMAIL PROTECTED]> wrote: > > > > Hi, > > > > There are two sunchronization points: on the stream and on the reader. > > Using > > different FSDirectoriy and IndexReaders should solve this. I'll let you > > know > > once I code it. Right now I'm checking if making my Documents store less > > data will move the bottleneck to some other place. > > > > Thanks again, > > Oren Shir > > > > On 11/21/05, Doug Cutting <[EMAIL PROTECTED]> wrote: > > > > > > Jay Booth wrote: > > > > I had a similar problem with threading, the problem turned out to be > > > that in > > > > the back end of the FSDirectory class I believe it was, there was a > > > > synchronized block on the actual RandomAccessFile resource when > > reading > > > a > > > > block of data from it... high-concurrency situations caused threads to > > > stack > > > > up in front of this synchronized block and our CPU time wound up being > > > spent > > > > thrashing between blocked threads instead of doing anything useful. > > > > > > This is correct. In Lucene, multiple streams per file are created by > > > cloning, and all clones of an FSDirectory input stream share a > > > RandomAccessFile and must synchronize input from it. MmapDirectory does > > > not have this limitation. If your indexes are less than a few GB or you > > > are using 64-bit hardware, then MmapDirectory should work well for you. > > > Otherwise it would be simple to write an nio-based Directory that does > > > not use mmap that is also unsynchronized. Such a contribution would be > > > welcome. > > > > > > > Making multiple IndexSearchers and FSDirectories didn't help because > > in > > > the > > > > back end, lucene consults a singleton HashMap of some kind (don't > > > remember > > > > implementation) that maintained a single FSDirectory for any given > > index > > > > being accessed from the JVM... multiple calls to > > > FSDirectory.getDirectory > > > > actually return the same FSDirectory object with synchronization at > > the > > > same > > > > point. > > > > > > This does not make sense to me. FSDirectory does keep a cache of > > > FSDirectory instances, but i/o should not be synchronized on these. One > > > should be able to open multiple input streams on the same file from an > > > FSDirectory. But this would not be a great solution, since file handle > > > limits would soon become a problem. > > > > > > Doug > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
