Hi Roman, If you're looking for regression testing then https://github.com/sul-dlss/rspec-solr might be worth looking at. If you're not a ruby shop, doing something similar in another language shouldn't be to hard.
The basic idea is that you setup a set of tests like "If the query is X, then the document with id Y should be in the first 10 results" "If the query is S, then a document with title T should be the first result" "If the query is P, then a document with author Q should not be in the first 10 result" and that you run these whenever you tune your scoring formula to ensure that you haven't introduced unintended effects. New ideas/requirements for your relevance ranking should always result in writing new tests - that will probably fail until you tune your scoring formula. This is certainly no magic bullet, but it will give you some confidence that you didn't make things worse. And - in my humble opinion - it also gives you the benefit of discouraging you from tuning your scoring just for fun. To put it bluntly: if you cannot write up a requirement in form of a test, you probably have no need to tune your scoring. Regards, -- Steffen On Tuesday, February 12, 2013 at 23:03 , Roman Chyla wrote: > Hi, > I do realize this is a very broad question, but still I need to ask it. > Suppose you make a change into the scoring formula. How do you > test/know/see what impact it had? Any framework out there? > > It seems like people are writing their own tools to measure relevancy. > > Thanks for any pointers, > > roman
