Re: Question about how to speed up custom scoring

2009-10-11 Thread scott w
On Sun, Oct 11, 2009 at 9:10 AM, Jake Mannix wrote: > What do you mean "not something I can plug in on top of my original query"? > > Do you mean that you can't do it like the more complex example in the class > you posted earlier in the thread, where you take a linear combination of > the > Map

Re: Question about how to speed up custom scoring

2009-10-11 Thread Jake Mannix
What do you mean "not something I can plug in on top of my original query"? Do you mean that you can't do it like the more complex example in the class you posted earlier in the thread, where you take a linear combination of the Map -based score, and the regular text score? Another option is to j

Re: Question about how to speed up custom scoring

2009-10-10 Thread scott w
Haven't tried it yet but looking at it closer it looks like it's not something I can plug in on top of my original query. I am definitely happy using an approximation for the sake of performance but I do need to be able to have the original results stay the same. On Fri, Oct 9, 2009 at 5:32 PM, Ja

Re: Question about how to speed up custom scoring

2009-10-09 Thread Jake Mannix
Great Scott (hah!) - please do report back, even if it just works fine and you have no more questions, I'd like to know whether this really is what you were after and actually works for you. Note that the FieldCache is kinda "magic" - it's lazy (so the first query will be slow and you should fire

Re: Question about how to speed up custom scoring

2009-10-09 Thread scott w
Thanks Jake! I will test this out and report back soon in case it's helpful to others. Definitely appreciate the help. Scott On Fri, Oct 9, 2009 at 3:33 PM, Jake Mannix wrote: > On Fri, Oct 9, 2009 at 3:07 PM, scott w wrote: > > > Example Document: > > model_1_score = 0.9 > > model_2_score = 0

Re: Question about how to speed up custom scoring

2009-10-09 Thread Jake Mannix
On Fri, Oct 9, 2009 at 3:07 PM, scott w wrote: > Example Document: > model_1_score = 0.9 > model_2_score = 0.3 > model_3_score = 0.7 > > I want to be able to pass in the following map at query time: > {model_1_score=0.4, model_2_score=0.7} and have that map get used as input > to a custom score f

Re: Question about how to speed up custom scoring

2009-10-09 Thread scott w
Hi Jake -- Sorry for the confusion. I have two similar but slightly different use cases in mind and the example I gave you corresponds to one use case while the code corresponds to the other slightly more complicated one. Ignore the original example, and let me restate the one I have in mind so it

Re: Question about how to speed up custom scoring

2009-10-09 Thread Jake Mannix
Hey Scott, I'm still not sure I understand what your dynamic boosts are for: they are the names of fields, right, not terms in the fields? So in terms of your example { company = microsoft, city = redmond, size = big }, the three possible choices for keys in your map are company, city, or size,

Re: Question about how to speed up custom scoring

2009-10-09 Thread scott w
(Apologies if this message gets sent more than once. I received an error sending it the first two times so sent directly to Jake but reposting to group.) Hi Jake -- Thanks for the feedback. What I am trying to implement is a way to custom score documents using a scoring function that takes as inp

Re: Question about how to speed up custom scoring

2009-10-09 Thread scott w
Right exactly. I looked into payload initially and realized it wouldn't work for my use case. On Fri, Oct 9, 2009 at 2:00 PM, Grant Ingersoll wrote: > Oops, just reread and realized you wanted query time weights. Payloads are > an index time thing. > > > On Oct 9, 2009, at 5:49 PM, Grant Ingers

Re: Question about how to speed up custom scoring

2009-10-09 Thread Grant Ingersoll
Oops, just reread and realized you wanted query time weights. Payloads are an index time thing. On Oct 9, 2009, at 5:49 PM, Grant Ingersoll wrote: If you are trying to add specific term weights to terms in the index and then incorporate them into scoring, you might benefit from payloads a

Re: Question about how to speed up custom scoring

2009-10-09 Thread Grant Ingersoll
If you are trying to add specific term weights to terms in the index and then incorporate them into scoring, you might benefit from payloads and the PayloadTermQuery option. See http://www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads/ -Grant On Oct 8, 2009, at 11:56 AM

Re: Question about how to speed up custom scoring

2009-10-09 Thread Jake Mannix
Scott, To reiterate what Erick and Andrzej's said: calling IndexReader.document(docId) in your inner scoring loop is the source of your performance problem - iterating over all these stored fields is what is killing you. To do this a better way, can you try to explain exactly what this Scorer

Re: Question about how to speed up custom scoring

2009-10-09 Thread scott w
Thanks for the suggestions Erick. I am using Lucene 2.3. Terms are stored and given Andrzej's comments in the follow up email sounds like it's not the stored field issue. I'll keep investigating... thanks, Scott On Thu, Oct 8, 2009 at 8:06 AM, Erick Erickson wrote: > I suspect your problem here

Re: Question about how to speed up custom scoring

2009-10-08 Thread Andrzej Bialecki
Erick Erickson wrote: I suspect your problem here is the line: document = indexReader.document( doc ); See the caution in the docs You could try using lazy loading (so you don't load all the terms of the document, just those you're interested in). And I *think* (but it's been a while) that if t

Re: Question about how to speed up custom scoring

2009-10-08 Thread Erick Erickson
I suspect your problem here is the line: document = indexReader.document( doc ); See the caution in the docs You could try using lazy loading (so you don't load all the terms of the document, just those you're interested in). And I *think* (but it's been a while) that if the terms you load are in

Re: Question about how to speed up custom scoring

2009-10-08 Thread scott w
Oops, forgot to include the class I mentioned. Here it is: public class QueryTermBoostingQuery extends CustomScoreQuery { private Map queryTermWeights; private float bias; private IndexReader indexReader; public QueryTermBoostingQuery( Query q, Map termWeights, IndexReader indexReader, fl

Question about how to speed up custom scoring

2009-10-08 Thread scott w
I am trying to come up with a performant query that will allow me to use a custom score where the custom score is a sum-product over a set of query time weights where each weight gets applied only if the query time term exists in the document . So for example if I have a doc with three fields: comp