RE: FieldsReader synchronized access vs. ThreadLocal ?

2006-05-19 Thread Robert Engels
Given that Lucene is generally VERY CPU bound, having stalled processors implies that those threads (and more) are blocked, either by IO, or by a synchronized block - as long as you have more threads than processors. If the machine has a POOR disk subsystem in comparison to the CPU speed, and the

RE: FieldsReader synchronized access vs. ThreadLocal ?

2006-05-19 Thread Robert Engels
As an aside, On my VERY crappy 1.2 ghz single CPU P4, using a index of 300k documents, I can perform 50 searches per second (returning 20 document matches each). This includes the time to serialize and send the results to the client (although the client is on the same machine, but it also competes

Re: FieldsReader synchronized access vs. ThreadLocal ?

2006-05-19 Thread Peter Keegan
The queries are mostly boolean (all AND'd terms), no. of terms varies anywhere from a few to 25 or more, with 1 or 2 sort fields. My tests are designed to measure total query throughput, not just raw search speed. The client test program blasts queries from >50 threads over a socket and runs on a

RE: FieldsReader synchronized access vs. ThreadLocal ?

2006-05-19 Thread Robert Engels
If you can get much higher throughput from a single threaded client, then you should queue the requests in the server and process them from a single thread (or a small pool of threads). If you can get much higher throughput from a single threaded client seems to also imply that Lucene (or at least

Re: FieldsReader synchronized access vs. ThreadLocal ?

2006-05-19 Thread Yonik Seeley
On 5/19/06, Peter Keegan <[EMAIL PROTECTED]> wrote: The client test program blasts queries from >50 threads over a socket and runs on a separate server from Lucene. I can get much higher rates by just blasting from a single thread in the client, but this doesn't simulate the real use model. Wow

Re: FieldsReader synchronized access vs. ThreadLocal ?

2006-05-19 Thread Peter Keegan
The use model is to be able to handle bursts of heavy query load with acceptable latency. The test program's threads send requests continuously with a 50 msec delay in between. A separate thread reads all the results. This is actually much harsher than the expected load and the latency is high, bu

[jira] Created: (LUCENE-574) support for vjc java compiler, also known as J#

2006-05-19 Thread George J. Carrette (JIRA)
support for vjc java compiler, also known as J# --- Key: LUCENE-574 URL: http://issues.apache.org/jira/browse/LUCENE-574 Project: Lucene - Java Type: New Feature Components: Other Versions: 2.0 Environment: Mi

[jira] Updated: (LUCENE-574) support for vjc java compiler, also known as J#

2006-05-19 Thread George J. Carrette (JIRA)
[ http://issues.apache.org/jira/browse/LUCENE-574?page=all ] George J. Carrette updated LUCENE-574: -- Attachment: lucene-2.0-vjc.zip This is has the "vjc" folder for compiling the Java Lucene sources with the J# compiler. It also has a couple files t

[jira] Commented: (LUCENE-561) ParallelReader fails on deletes and on seeks of previously unused fields

2006-05-19 Thread Yonik Seeley (JIRA)
[ http://issues.apache.org/jira/browse/LUCENE-561?page=comments#action_12412532 ] Yonik Seeley commented on LUCENE-561: - This patch also fixes another little bug... the previous behavior bypassed synchronization of deleteDocument in subreaders. > Para

RE: FieldsReader synchronized access vs. ThreadLocal ?

2006-05-19 Thread Robert Engels
fyi, when the OS disk cache is not primed, I get better performance from more threads (since when one thread is blocked on IO, the other may find its data available, and continue the query), but as the cache becomes "ready" the more threads I use the worse the performance (due to the overhead of co

[jira] Resolved: (LUCENE-561) ParallelReader fails on deletes and on seeks of previously unused fields

2006-05-19 Thread Yonik Seeley (JIRA)
[ http://issues.apache.org/jira/browse/LUCENE-561?page=all ] Yonik Seeley resolved LUCENE-561: - Fix Version: 2.0 Resolution: Fixed Assign To: Yonik Seeley Patch applied, thanks Chuck! > ParallelReader fails on deletes and on seeks of pr

[jira] Resolved: (LUCENE-556) MatchAllDocsQuery, MultiSearcher and a custom HitCollector throwing exception

2006-05-19 Thread Yonik Seeley (JIRA)
[ http://issues.apache.org/jira/browse/LUCENE-556?page=all ] Yonik Seeley resolved LUCENE-556: - Resolution: Fixed Assign To: Yonik Seeley Fixed. Added empty implementation for MatchAllDocsQuery and ConstantScoreQuery. > MatchAllDocsQuery, Mul

Re: Changing Lucene scoring?

2006-05-19 Thread karl wettin
On Mon, 2006-05-08 at 18:34 -0700, Otis Gospodnetic wrote: > about the possibility of eliminating floating point calculations from > Lucene's scoring Did you look in to this? - To unsubscribe, e-mail: [EMAIL PROTECTED] For add

Re: Vector

2006-05-19 Thread karl wettin
On Sat, 2006-05-13 at 01:38 +0200, karl wettin wrote: > here are my comments. It seems to run fine on Lists. But I have only been running it on an issue 550 index, so I have no idea how the Directory implementation index will take it. No test case, no nothing. Just tried it in live environment fo

Re: Changing Lucene scoring?

2006-05-19 Thread Otis Gospodnetic
On Mon, 2006-05-08 at 18:34 -0700, Otis Gospodnetic wrote: > about the possibility of eliminating floating point calculations from > Lucene's scoring Did you look in to this? No, not yet. I did start looking at how exactly scoring works (debugger, step through), but then had to switch to ben