Note that this can lead to performance issues. Queries with lots of hits require lots of scoring and this will make queries slower. We had this case with a client about 2 weeks ago. We were able to spot this change in the change in the average number of hits before/after query changes (meant to help with relevance) using Sematext Search Analytics which clearly shows a spike in the number of queries with lots of hits (very "loose" queries) and we were able to correlate that with performance issues.
So careful with expanding your queries too much without watching before/after performance. Otis -- Solr & ElasticSearch Support -- http://sematext.com/ Performance Monitoring -- http://sematext.com/spm On Tue, Jul 23, 2013 at 3:52 PM, Jack Krupansky <j...@basetechnology.com> wrote: > One classic approach is to simply use the full text of the suspect text as > well as bigrams and trigrams (phrases) from that text with "OR" operators. > The top results will be the documents that most closely "match" the subject > text. That provides a visual set similar results. You will then have to > apply some heuristic of your own as far as how many top results to look at > or what score to cut off at. The use of "OR" operators assures that similar > documents will be found even if not 100% of the words are used. Yes, "OR" > guarantees that your total result count will be high, but scoring assures > that the top results will be more relevant. > > -- Jack Krupansky > > -----Original Message----- From: Furkan KAMACI > Sent: Tuesday, July 23, 2013 6:16 AM > To: solr-user@lucene.apache.org > Subject: Re: Document Similarity Algorithm at Solr/Lucene > > Actually I need a specialized algorithm. I want to use that algorithm to > detect duplicate blog posts. > > 2013/7/23 Tommaso Teofili <tommaso.teof...@gmail.com> > >> Hi, >> >> I you may leverage and / or improve MLT component [1]. >> >> HTH, >> Tommaso >> >> [1] : http://wiki.apache.org/solr/MoreLikeThis >> >> >> 2013/7/23 Furkan KAMACI <furkankam...@gmail.com> >> >> > Hi; >> > >> > Sometimes a huge part of a document may exist in another document. As >> like >> > in student plagiarism or quotation of a blog post at another blog post. >> > Does Solr/Lucene or its libraries (UIMA, OpenNLP, etc.) has any class to >> > detect it? >> > >> >
