Sounds exactly like our tool Quepid http://quepid.com :) which is our test driven search toolbox.
Whether or not Quepid is the right fit for your application, we advocate for a style of work called Test-Driven Relevancy. http://opensourceconnections.com/blog/2013/10/14/what-is-test-driven-search-relevancy/ I've used both functional tests through Quepid and for clients that are very particular or have the right analytics data judgement-list based tests. Judgement lists are basically expert-graded ratings of results. The decision when to use which really depends on the other side of the table. Who is defining search correctness? How is it defined? Some organizations have a lot of infrastructure to do this, and can give you judgement data to tell you what the right answer is. Other organizations have much softer requirements and are not as experienced with search. Sometimes ad-hoc, assertion based testing is right for them. Really the biggest question is how you define search correctness. Search is an unique a piece of the user experience as any other part of the application. You likely don't have the right expertise to define what relevant means for your application. So a big part of our view of test-driven relevancy is collaborating with domain/content/user experts that can help define the right experience. Shoot me an email if you want to chat at some point, Cheers, -Doug On Wed, Sep 30, 2015 at 9:58 AM, marotosg <marot...@gmail.com> wrote: > Hi, > > I have some doubts about how to define a process to evaluate the quality of > search results. > I have a solr collection with 4M documents with information about people. I > search across several fields like first name ,second name, email, > address, > phone etc. > > There is plenty of logic in the query. Some fields ranks higher, exact > match > ranks higher than trail search etc. > > I was thinking on an approach where I can create some automated tests. > Based > on some searches if the results are good enough or the ones which come > first > are actually better than the other ones. > I would like to be able to identify how new functionality affects previous > results. > > I thought on this creating two type of tests. > a) Functional tests which will test if the functionality is correct. They > will be based on a subset of records which are static. In this case, I know > what order results should come back from the query. > > b) Based on full data. I would like to run queries and see if the results > are good enough. That's the part I am not sure if makes sense or how to do > it. > > I am not sure if that's not correct or if there is a any standad to follow. > Any help would be much appreciate. > > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Regression-tests-and-evaluate-quality-of-results-tp4232047.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- *Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections <http://opensourceconnections.com>, LLC | 240.476.9983 Author: Relevant Search <http://manning.com/turnbull> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
