Re: What are the options for obtaining IDF at interactive speeds?

2013-07-10 Thread Kathryn Mazaitis
I didn't try indexing each term as a separate document (and if I had I probably would've just used tv.tf_idf instead of a functional query -- why not?). The regular functional query which required sending a separate request for each of thousands of terms was wy dominated by the overhead of

Re: What are the options for obtaining IDF at interactive speeds?

2013-07-08 Thread Kathryn Mazaitis
Hi All, Resolution: I ended up cheating. :P Though now that I look at it, I think this was Roman's second suggestion. Thanks! Since the application that will be processing the IDF figures is located on the same machine as SOLR, I opened a second IndexReader on the lucene index and used

Re: What are the options for obtaining IDF at interactive speeds?

2013-07-08 Thread Roman Chyla
Hi, I am curious about the functional query, did you try it and it didn't work? or was it too slow? idf(other_field,field(term)) Thanks! roman On Mon, Jul 8, 2013 at 4:34 PM, Kathryn Mazaitis ka...@rivard.org wrote: Hi All, Resolution: I ended up cheating. :P Though now that I look at

Re: What are the options for obtaining IDF at interactive speeds?

2013-07-03 Thread Mikhail Khludnev
Katie, This case is actually really hard to get. Just let me provide the contra-sample, to let you explain problem better by spotting the gap. What if I say that, debugQuery=true provides tf, idf for the terms and documents from the requested page of results. Why you can't use explain to solve

Re: What are the options for obtaining IDF at interactive speeds?

2013-07-03 Thread Roman Chyla
Hi Kathryn, I wonder if you could index all your terms as separate documents and then construct a new query (2nd pass) q=term:term1 OR term:term2 OR term:term3 and use func to score them *idf(other_field,field(term))* * * the 'term' index cannot be multi-valued, obviously. Other than that, if

What are the options for obtaining IDF at interactive speeds?

2013-07-02 Thread Kathryn Mazaitis
Hi, I'm using SOLRJ to run a query, with the goal of obtaining: (1) the retrieved documents, (2) the TF of each term in each document, (3) the IDF of each term in the set of retrieved documents (TF/IDF would be fine too) ...all at interactive speeds, or 10s per query. This is a demo, so if all