Re: SloppyPhraseScorer behavior change
Moreover just checked .. autoGeneratePhraseQueries=true is set for both 3.4 and 4.0 in my schema. Thanks Varun On Fri, Jan 11, 2013 at 1:04 PM, varun srivastava varunmail...@gmail.comwrote: Hi Jack, Is this a new change done in solr 4.0 ? Seems autoGeneratePhraseQueries option is present from solr 3.1. Just wanted to confirm this is the difference causing change in behavior between 3.4 and 4.0. Thanks Varun On Mon, Dec 24, 2012 at 3:00 PM, Jack Krupansky j...@basetechnology.comwrote: Thanks. Sloppy phrase requires that the query terms be in a phrase, but you don't have any quotes in your query. Depending on your schema field type you may be running into a change in how auto-generated phrase queries are handled. It used to be that apple0ipad would always be treated as the quoted phrase apple 0 ipad, but now that is only true if your field type has autoGeneratePhraseQueries=true set. Now, if you don't have that option set, the term gets treated as (apple OR 0 OR ipad), which is a lot looser than the exact phrase. Look at the new example schema for the text_en_splitting field type as an example. -- Jack Krupansky -Original Message- From: varun srivastava Sent: Monday, December 24, 2012 5:49 PM To: solr-user@lucene.apache.org Subject: Re: SloppyPhraseScorer behavior change Hi Jack, My query was simple /solr/select?query=ipad apple apple0ipad and doc contained apple ipad . If you see the patch attached with the bug 3215 , you will find following comment. I want to confirm whether the behaviour I am observing is in sync with what the patch developer intended or its just some regression bug. In solr 3.4 phrase order is honored, whereas in solr 4.0 phrase order is not honored, i.e. apple ipad and ipad apple both treated as same. /** + * Score a candidate doc for all slop-valid position-combinations (matches) + * encountered while traversing/hopping the PhrasePositions. + * br The score contribution of a match depends on the distance: + * br - highest score for distance=0 (exact match). + * br - score gets lower as distance gets higher. + * brExample: for query a b~2, a document x a b a y can be scored twice: + * once for a b (distance=0), and once for b a (distance=2). + * brPossibly not all valid combinations are encountered, because for efficiency + * we always propagate the least PhrasePosition. This allows to base on + * PriorityQueue and move forward faster. + * As result, for example, document a b c b a + * would score differently for queries a b c~4 and c b a~4, although + * they really are equivalent. + * Similarly, for doc a b c b a f g, query c b~2 + * would get same score as g f~2, although c b~2 could be matched twice. + * We may want to fix this in the future (currently not, for performance reasons). + */ On Mon, Dec 24, 2012 at 1:21 PM, Jack Krupansky j...@basetechnology.com **wrote: Could you post the full query URL, so we can see exactly what your query was? Or, post the output of debug=query, which will show us what Lucene query was generated. -- Jack Krupansky -Original Message- From: varun srivastava Sent: Monday, December 24, 2012 1:53 PM To: solr-user@lucene.apache.org Subject: SloppyPhraseScorer behavior change Hi, Due to following bug fix https://issues.apache.org/jira/browse/LUCENE-3215https://issues.apache.org/**jira/browse/LUCENE-3215 https:**//issues.apache.org/jira/**browse/LUCENE-3215https://issues.apache.org/jira/browse/LUCENE-3215observing a change in behavior of SloppyPhraseScorer. I just wanted to confirm my understanding with you all. After solr 3.5 ( bug is fixed in 3.5), if there is a document a b c d e, then in solr 3.4 only query a b will match with document, but in solr 3.5 onwards, both query a b and b a will match. Is it right ? Thanks Varun
Re: SloppyPhraseScorer behavior change
Could you post the full query URL, so we can see exactly what your query was? Or, post the output of debug=query, which will show us what Lucene query was generated. -- Jack Krupansky -Original Message- From: varun srivastava Sent: Monday, December 24, 2012 1:53 PM To: solr-user@lucene.apache.org Subject: SloppyPhraseScorer behavior change Hi, Due to following bug fix https://issues.apache.org/jira/browse/LUCENE-3215 observing a change in behavior of SloppyPhraseScorer. I just wanted to confirm my understanding with you all. After solr 3.5 ( bug is fixed in 3.5), if there is a document a b c d e, then in solr 3.4 only query a b will match with document, but in solr 3.5 onwards, both query a b and b a will match. Is it right ? Thanks Varun
Re: SloppyPhraseScorer behavior change
Hi Jack, My query was simple /solr/select?query=ipad apple apple0ipad and doc contained apple ipad . If you see the patch attached with the bug 3215 , you will find following comment. I want to confirm whether the behaviour I am observing is in sync with what the patch developer intended or its just some regression bug. In solr 3.4 phrase order is honored, whereas in solr 4.0 phrase order is not honored, i.e. apple ipad and ipad apple both treated as same. /** + * Score a candidate doc for all slop-valid position-combinations (matches) + * encountered while traversing/hopping the PhrasePositions. + * br The score contribution of a match depends on the distance: + * br - highest score for distance=0 (exact match). + * br - score gets lower as distance gets higher. + * brExample: for query a b~2, a document x a b a y can be scored twice: + * once for a b (distance=0), and once for b a (distance=2). + * brPossibly not all valid combinations are encountered, because for efficiency + * we always propagate the least PhrasePosition. This allows to base on + * PriorityQueue and move forward faster. + * As result, for example, document a b c b a + * would score differently for queries a b c~4 and c b a~4, although + * they really are equivalent. + * Similarly, for doc a b c b a f g, query c b~2 + * would get same score as g f~2, although c b~2 could be matched twice. + * We may want to fix this in the future (currently not, for performance reasons). + */ On Mon, Dec 24, 2012 at 1:21 PM, Jack Krupansky j...@basetechnology.comwrote: Could you post the full query URL, so we can see exactly what your query was? Or, post the output of debug=query, which will show us what Lucene query was generated. -- Jack Krupansky -Original Message- From: varun srivastava Sent: Monday, December 24, 2012 1:53 PM To: solr-user@lucene.apache.org Subject: SloppyPhraseScorer behavior change Hi, Due to following bug fix https://issues.apache.org/**jira/browse/LUCENE-3215https://issues.apache.org/jira/browse/LUCENE-3215observing a change in behavior of SloppyPhraseScorer. I just wanted to confirm my understanding with you all. After solr 3.5 ( bug is fixed in 3.5), if there is a document a b c d e, then in solr 3.4 only query a b will match with document, but in solr 3.5 onwards, both query a b and b a will match. Is it right ? Thanks Varun
Re: SloppyPhraseScorer behavior change
Thanks. Sloppy phrase requires that the query terms be in a phrase, but you don't have any quotes in your query. Depending on your schema field type you may be running into a change in how auto-generated phrase queries are handled. It used to be that apple0ipad would always be treated as the quoted phrase apple 0 ipad, but now that is only true if your field type has autoGeneratePhraseQueries=true set. Now, if you don't have that option set, the term gets treated as (apple OR 0 OR ipad), which is a lot looser than the exact phrase. Look at the new example schema for the text_en_splitting field type as an example. -- Jack Krupansky -Original Message- From: varun srivastava Sent: Monday, December 24, 2012 5:49 PM To: solr-user@lucene.apache.org Subject: Re: SloppyPhraseScorer behavior change Hi Jack, My query was simple /solr/select?query=ipad apple apple0ipad and doc contained apple ipad . If you see the patch attached with the bug 3215 , you will find following comment. I want to confirm whether the behaviour I am observing is in sync with what the patch developer intended or its just some regression bug. In solr 3.4 phrase order is honored, whereas in solr 4.0 phrase order is not honored, i.e. apple ipad and ipad apple both treated as same. /** + * Score a candidate doc for all slop-valid position-combinations (matches) + * encountered while traversing/hopping the PhrasePositions. + * br The score contribution of a match depends on the distance: + * br - highest score for distance=0 (exact match). + * br - score gets lower as distance gets higher. + * brExample: for query a b~2, a document x a b a y can be scored twice: + * once for a b (distance=0), and once for b a (distance=2). + * brPossibly not all valid combinations are encountered, because for efficiency + * we always propagate the least PhrasePosition. This allows to base on + * PriorityQueue and move forward faster. + * As result, for example, document a b c b a + * would score differently for queries a b c~4 and c b a~4, although + * they really are equivalent. + * Similarly, for doc a b c b a f g, query c b~2 + * would get same score as g f~2, although c b~2 could be matched twice. + * We may want to fix this in the future (currently not, for performance reasons). + */ On Mon, Dec 24, 2012 at 1:21 PM, Jack Krupansky j...@basetechnology.comwrote: Could you post the full query URL, so we can see exactly what your query was? Or, post the output of debug=query, which will show us what Lucene query was generated. -- Jack Krupansky -Original Message- From: varun srivastava Sent: Monday, December 24, 2012 1:53 PM To: solr-user@lucene.apache.org Subject: SloppyPhraseScorer behavior change Hi, Due to following bug fix https://issues.apache.org/**jira/browse/LUCENE-3215https://issues.apache.org/jira/browse/LUCENE-3215observing a change in behavior of SloppyPhraseScorer. I just wanted to confirm my understanding with you all. After solr 3.5 ( bug is fixed in 3.5), if there is a document a b c d e, then in solr 3.4 only query a b will match with document, but in solr 3.5 onwards, both query a b and b a will match. Is it right ? Thanks Varun
