I agree! A a matter of fact, that is exactly what I just wrote here: http://www.jroller.com/otis/entry/followup_open_relevance_project#comment-1242703187000
"....For example, couldn't a vendor use it to compare old implementation to new implementation and provide some kind of metric showing improvements in new version?...." The first "vendor" in ORP's case might be Lucene. My hope would be that others could and would take what ORP builds and apply it to their implementations. My next wish after that would be to see others publish the results. But, I think we'll never see any results from commercial vendors - I have a feeling they don't have much to gain by exposing their results to the competition and to the public. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: Ted Dunning <ted.dunn...@gmail.com> > To: general@lucene.apache.org > Sent: Monday, May 18, 2009 11:12:41 PM > Subject: Re: Open Relevance Project? > > I completely agee with this. In practice, search engines and to a larger > extent recommendation engines shape user behavior and are, in turn, shaped > by user behavior so that static relevancy tests are of only very limited > value in the end game. > > But it is still *very* nice to have them. > > On Mon, May 18, 2009 at 8:00 PM, Mark Miller wrote: > > > Grant Ingersoll wrote: > > > >> Some interesting discussion at > >> > http://thenoisychannel.com/2009/05/18/copying-trec-is-the-wrong-track-for-the-enterprise/ > >> > > That was an interesting read. I think a lot of the argument misses the > > point. It doesn't seem to me that the main benefit or intent comes from > > 'bake offs' with other search engines ("Selling search applications to > > enterprises isn't, in my experience, about winning relevance bake-offs.") - > > the main benefit is allowing us to measure changes and improvements to > > Lucene's relevancy calculations and to make judgments about how Lucene > > currently performs. I see it easily as important as the Lucene benchmark > > contrib. Its not going to be a secret sauce, just like the benchmarker has > > been no secret sauce - but its going to make it easier to reliably improve > > Lucene in the future. > > > >
