Re: TermInfosReader lazy term index reading

2007-02-02 Thread Yonik Seeley
FYI, I filed a Solr bug for this issue: https://issues.apache.org/jira/browse/SOLR-138 -Yonik - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]

Re: TermInfosReader lazy term index reading

2007-02-02 Thread robert engels
I think that is much more involved... I don't think there is an easy way to move a query between threads/pools once it has started unless you restart the entire query. You might be able to dynamically lower the thread priority however when you detect a long query, so that smaller (faster) q

Re: TermInfosReader lazy term index reading

2007-02-02 Thread Doron Cohen
robert engels <[EMAIL PROTECTED]> wrote on 02/02/2007 14:08:46: > You might be able to quantify the search request ahead of time (# of > terms, # of high frequency terms, etc.) and assign the request to the > appropriate pool (quick, normal, lengthy). > > Then you can assign an appropriate # of th

Re: TermInfosReader lazy term index reading

2007-02-02 Thread robert engels
You might be able to quantify the search request ahead of time (# of terms, # of high frequency terms, etc.) and assign the request to the appropriate pool (quick, normal, lengthy). Then you can assign an appropriate # of threads to each pool. Most people understand that complex queries migh

Re: TermInfosReader lazy term index reading

2007-02-02 Thread Yonik Seeley
On 2/2/07, robert engels <[EMAIL PROTECTED]> wrote: For a process that is mostly CPU bound (which is the case with Lucene if the index is in the OS cache), having so many "active" threads will actually hurt performance due to the context switching and synchronization. Sure... it certainly wasn'

Re: TermInfosReader lazy term index reading

2007-02-02 Thread robert engels
FYI, For a process that is mostly CPU bound (which is the case with Lucene if the index is in the OS cache), having so many "active" threads will actually hurt performance due to the context switching and synchronization. Better to use a request queue / thread pool. (I think I read somewh

Re: TermInfosReader lazy term index reading

2007-02-02 Thread Doug Cutting
Yonik Seeley wrote: I think synchronization can become expensive under heavy contention, regardless of how lightweight the code inside. I'm skeptical of this. It's possible, but I've never seen it. Doug - To unsubscribe, e-m

Re: TermInfosReader lazy term index reading

2007-02-02 Thread Yonik Seeley
On 2/2/07, Doug Cutting <[EMAIL PROTECTED]> wrote: Yonik Seeley wrote: > I ran across a situation where a great number of threads were blocked on > ensureIndexIsRead(), even after it had already been loaded. That sounds bizarre. A sync block that tests a field for non-null shouldn't tie things

Re: TermInfosReader lazy term index reading

2007-02-02 Thread Doug Cutting
Yonik Seeley wrote: I ran across a situation where a great number of threads were blocked on ensureIndexIsRead(), even after it had already been loaded. That sounds bizarre. A sync block that tests a field for non-null shouldn't tie things up much, I wouldn't think. Are you sure that one of

Re: TermInfosReader lazy term index reading

2007-02-02 Thread Yonik Seeley
On 2/2/07, Doug Cutting <[EMAIL PROTECTED]> wrote: Yonik Seeley wrote: > What was the use-case behind loading the term index lazily? > I'm having a hard time figuring out what one would do with an > IndexReader that doesn't involve a term lookup somehow. Index merging only iterates through terms

Re: TermInfosReader lazy term index reading

2007-02-02 Thread Otis Gospodnetic
Check index version? Otis - Original Message From: Yonik Seeley <[EMAIL PROTECTED]> To: java-dev@lucene.apache.org Sent: Friday, February 2, 2007 1:36:35 PM Subject: TermInfosReader lazy term index reading What was the use-case behind loading the term index lazily? I'm ha

Re: TermInfosReader lazy term index reading

2007-02-02 Thread robert engels
You only need to load it for segments that are read, instead of paying the init price on all segments that may never be used. On Feb 2, 2007, at 12:36 PM, Yonik Seeley wrote: What was the use-case behind loading the term index lazily? I'm having a hard time figuring out what one would do with

Re: TermInfosReader lazy term index reading

2007-02-02 Thread Doug Cutting
Yonik Seeley wrote: What was the use-case behind loading the term index lazily? I'm having a hard time figuring out what one would do with an IndexReader that doesn't involve a term lookup somehow. Index merging only iterates through terms. Doug ---

TermInfosReader lazy term index reading

2007-02-02 Thread Yonik Seeley
What was the use-case behind loading the term index lazily? I'm having a hard time figuring out what one would do with an IndexReader that doesn't involve a term lookup somehow. -Yonik - To unsubscribe, e-mail: [EMAIL PROTECTED]