All, Thank you for your comments and links, I will explore them.
I think that many people are facing similar questions - when they tune their search engines. Especially in Solr/Lucene community. While the requirements will be different, ultimately it is what they can do w lucene/solr that guides such efforts. As an example, let me use this https://github.com/romanchyla/r-ranking-fun/blob/master/plots/raw/test-plot-showing-factors.pdf?raw=true The graph shows you the effect of different values of qf parameter. This usecase is probably very common, so somebody already had probably done st similar In the real world, I would like to: 1) change something, 2) collect (clicks) data 3) apply statistical test (of my choice) to see if changes had the effect (be it worse or better) and see if that change is statistically significant. But do we have to write these tools from scratch again? All your comments are very valuable and useful. But I am still wondering if there are more tools one could use to tune the search. More comments welcome! Thank you! roman On Wed, Feb 13, 2013 at 1:04 PM, Amit Nithian <anith...@gmail.com> wrote: > Ultimately this is dependent on what your metrics for success are. For some > places it may be just raw CTR (did my click through rate increase) but for > other places it may be a function of money (either it may be gross revenue, > profits, # items sold etc). I don't know if there is a generic answer for > this question which is leading those to write their own frameworks b/c it's > very specific to your needs. A scoring change that leads to an increase in > CTR may not necessarily lead to an increase in the metric that makes your > business go. > > > On Tue, Feb 12, 2013 at 10:31 PM, Steffen Elberg Godskesen < > steffen.godske...@gmail.com> wrote: > > > > > Hi Roman, > > > > If you're looking for regression testing then > > https://github.com/sul-dlss/rspec-solr might be worth looking at. If > > you're not a ruby shop, doing something similar in another language > > shouldn't be to hard. > > > > > > The basic idea is that you setup a set of tests like > > > > "If the query is X, then the document with id Y should be in the first 10 > > results" > > "If the query is S, then a document with title T should be the first > > result" > > "If the query is P, then a document with author Q should not be in the > > first 10 result" > > > > and that you run these whenever you tune your scoring formula to ensure > > that you haven't introduced unintended effects. New ideas/requirements > for > > your relevance ranking should always result in writing new tests - that > > will probably fail until you tune your scoring formula. This is certainly > > no magic bullet, but it will give you some confidence that you didn't > make > > things worse. And - in my humble opinion - it also gives you the benefit > of > > discouraging you from tuning your scoring just for fun. To put it > bluntly: > > if you cannot write up a requirement in form of a test, you probably have > > no need to tune your scoring. > > > > > > Regards, > > > > -- > > Steffen > > > > > > > > On Tuesday, February 12, 2013 at 23:03 , Roman Chyla wrote: > > > > > Hi, > > > I do realize this is a very broad question, but still I need to ask it. > > > Suppose you make a change into the scoring formula. How do you > > > test/know/see what impact it had? Any framework out there? > > > > > > It seems like people are writing their own tools to measure relevancy. > > > > > > Thanks for any pointers, > > > > > > roman > > > > > > >
