Re: Improving TimeLimitedCollector

2009-06-27 Thread Mark Harwood
Odd. I see you're responding to a message from Shai I didn't get. Some mail being dropped somewhere along the line.. Why don't you use Thread.interrupt(), .isInterrupted() ? Not sure where exactly you mean for that? I'm not sure I understand that - how can a thread run >1 activity simult

Re: Improving TimeLimitedCollector

2009-06-27 Thread Earwin Burrfoot
Why don't you use Thread.interrupt(), .isInterrupted() ? On Sat, Jun 27, 2009 at 16:16, Shai Erera wrote: >> A downside of breaking it out into static methods like this is that a >> thread cannot run >1 time-limited activity simultaneously but I guess that >> might be a reasonable restriction. > >

Re: Improving TimeLimitedCollector

2009-06-27 Thread Shai Erera
> > A downside of breaking it out into static methods like this is that a > thread cannot run >1 time-limited activity simultaneously but I guess that > might be a reasonable restriction. > I'm not sure I understand that - how can a thread run >1 activity simultaneously anyway, and how's your impl

Re: Improving TimeLimitedCollector

2009-06-27 Thread Mark Harwood
Thanks for the feedback, Shai. So I guess you're suggesting breaking this out into a general utility class e.g. something like: class TimeLimitedThreadActivity { //called by client public static void startTimeLimitedActivity(long maxTimePermitted). public static voi

Re: Improving TimeLimitedCollector

2009-06-27 Thread Shai Erera
I like the overall approach. However it's very local to an IndexReader. I.e., if someone wanted to limit other operations (say indexing), or does not use an IndexReader (for a Scorer impl maybe), one cannot reuse it. What if we factor out the timeout logic to a Timeout class (I think it can be a s

Re: Improving TimeLimitedCollector

2009-06-26 Thread Mark Harwood
Going back to my post re TimeLimitedIndexReaders - here's an incomplete but functional prototype: http://www.inperspective.com/lucene/TimeLimitedIndexReader.java http://www.inperspective.com/lucene/TestTimeLimitedIndexReader.java The principle is that all reader accesses check a volatile vari

Re: Improving TimeLimitedCollector

2009-06-25 Thread Shai Erera
When this thread was focused on Scorers only, I wanted to offer the following: * Have a TimeLimitingQuery which will wrap another Query, and delegate all the calls to it, except queryWeight, which will create TimeLimitingWeight. * Have a TimeLimitingWeight which will wrap the original query's Quer

Re: Improving TimeLimitedCollector

2009-06-24 Thread Jason Rutherglen
This would be good however how would we obtain the thread? I believe this would require using a ThreadLocalish type of system which could be quite slow (to obtain the thread and lookup in the hashmap). One implementation I looked at before was to add this check in IndexReader.isDeleted (by overrid

Re: Improving TimeLimitedCollector

2009-06-24 Thread Mark Harwood
I think the Collector approach makes the most sense to me, since it's the only object I fully control in the search process. I cannot control Query implementations, and I cannot control the decisions made by IndexSearcher. But I can always wrap someone else's Collector with TLC and pass it

Re: Improving TimeLimitedCollector

2009-06-24 Thread Jason Rutherglen
> > > that should be idealy user decision > > > > ____ > > From: Shai Erera > > To: java-dev@lucene.apache.org > > Sent: Wednesday, 24 June, 2009 10:55:50 > > Subject: Re: Improving TimeLimitedCollector > > > >

Re: Improving TimeLimitedCollector

2009-06-24 Thread Earwin Burrfoot
y user decision > > > From: Shai Erera > To: java-dev@lucene.apache.org > Sent: Wednesday, 24 June, 2009 10:55:50 > Subject: Re: Improving TimeLimitedCollector > > But TimeLimitingCollector's logic is coded in its collect() method. The t

Re: Improving TimeLimitedCollector

2009-06-24 Thread eks dev
"if you need SOME results quickly than reduce this timeout" that should be idealy user decision From: Shai Erera To: java-dev@lucene.apache.org Sent: Wednesday, 24 June, 2009 10:55:50 Subject: Re: Improving TimeLimitedCollector But TimeLimitingCollec

Re: Improving TimeLimitedCollector

2009-06-24 Thread Shai Erera
But TimeLimitingCollector's logic is coded in its collect() method. The top scorer calls nextDoc() or advance() on all its sub-scorers, and only when a match is found it calls collect(). If we want the sub-scorers to check whether they should abort, we'd need to revamp (liked the word :)) TimeLimi

Improving TimeLimitedCollector

2009-06-23 Thread Jason Rutherglen
As we're revamping collectors, weights, and scorers, perhaps we can push time limiting into the individual subscorers? Currently on a boolean query, we're timing out the query at the top level which doesn't work well if the subqueries exceed the time limit.