RE: Analysis page output vs. actually getting search matches, a discrepency?
Um sorry for any confusion. I meant to say I solved my issue by inserting a charFilter before the WhitespaceTokenizerFactory to convert my problem word to a searchable form. I had a cut n paste malfunction below. Thanks guys. -Original Message- From: Robert Petersen [mailto:rober...@buy.com] Sent: Tuesday, July 19, 2011 11:06 AM To: solr-user@lucene.apache.org Subject: RE: Analysis page output vs. actually getting search matches, a discrepency? Thanks Eric, Unfortunately I'm stemming the same on both sides, similar to the SOLR example settings for the text type field. Default search field is moreWords, as I want yes. Since I don't have this problem for any other mfg names at all in our index of almost 10 mm product docs, and this shows that is should match in my best estimation. Note: LucidKStemFilterFactory does not take 'Sterling' down to 'Sterl' in indexing nor searching, it stays as 'Sterling'. I have given up on this. I've decided it is just an unexplainable anomaly, and have solved it by inserting a LucidKStemFilterFactory and just modifying that word to it's searchable form before hitting the WhitespaceTokenizerFactory, which is kind of hackish but solves my problem at least. This seller only has a couple hundred cheap products on our site, so I have bigger fish to fry at this point. I've wasted too much time trying to chase this down. Cheers all Robi -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Monday, July 18, 2011 5:33 PM To: solr-user@lucene.apache.org Subject: Re: Analysis page output vs. actually getting search matches, a discrepency? Hmmm, is there any chance that you're stemming one place and not the other? And I infer from your output that your default search field is "moreWords", is that true and expected? You might use luke or the TermsComponent to see what's actually in the index, I'm going to guess that you'll find "sterl" but not "sterling" as an indexed term and your problem is stemming, but that's a shot in the dark. Best Erick On Mon, Jul 18, 2011 at 5:37 PM, Robert Petersen wrote: > OK I did what Hoss said, it only confirms I don't get a match when I > should and that the query parser is doing the expected. Here are the > details for one test sku. > > My analysis page output is shown in my email starting this thread and > here is my query debug output. This absolutely should match but > doesn't. Both the indexing side and the query side are splitting on > case changes. This actually isn't a problem for any of our other > content, for instance there is no issue searching for 'VideoSecu'. > Their products come up fine in our searches regardless of casing in the > query. Only SterlingTek's products seem to be causing us issues. > > Indexed content has camel case, stored in the text field 'moreWords': > "SterlingTek's NB-2LH 2 Pack Batteries + Charger Combo for Canon DC301" > Search term not matching with camel case: "SterlingTek's" > Search term matching if no case changes: "Sterlingtek's" > > Indexing: > generateWordParts="1" > generateNumberParts="1" > catenateWords="1" > catenateNumbers="1" > catenateAll="0" > splitOnCaseChange="1" > preserveOriginal="0" > /> > Searching: > generateWordParts="1" > generateNumberParts="1" > catenateWords="0" > catenateNumbers="0" > catenateAll="0" > splitOnCaseChange="1" > preserveOriginal="0" > /> > > Thanks > > http://ssdevrh01.buy.com:8983/solr/1/select?indent=on&version=2.2&q= > SterlingTek%27s&fq=&start=0&rows=1&fl=*%2Cscore&qt=standard&wt=standard& > debugQuery=on&explainOther=sku%3A216473417&hl=on&hl.fl=&echoHandler=true > &adf > > > > 0 > 4 > name="handler">org.apache.solr.handler.component.SearchHandler > > sku:216473417 > on > true > > standard > on > 1 > 2.2 > *,score > on > 0 > SterlingTek's > standard > > > > > > > SterlingTek's > SterlingTek's > PhraseQuery(moreWords:"sterling tek") > moreWords:"sterling tek" > > sku:216473417 > > > 0.0 = fieldWeight(moreWords:"sterling tek" in 76351), product of: > 0.0 = tf(phraseFreq=0.0) > 19.502613 = idf(moreWords: sterling=1 tek=72) > 0.15625 = fieldNorm(field=moreWords, doc=7635
RE: Analysis page output vs. actually getting search matches, a discrepency?
Thanks Eric, Unfortunately I'm stemming the same on both sides, similar to the SOLR example settings for the text type field. Default search field is moreWords, as I want yes. Since I don't have this problem for any other mfg names at all in our index of almost 10 mm product docs, and this shows that is should match in my best estimation. Note: LucidKStemFilterFactory does not take 'Sterling' down to 'Sterl' in indexing nor searching, it stays as 'Sterling'. I have given up on this. I've decided it is just an unexplainable anomaly, and have solved it by inserting a LucidKStemFilterFactory and just modifying that word to it's searchable form before hitting the WhitespaceTokenizerFactory, which is kind of hackish but solves my problem at least. This seller only has a couple hundred cheap products on our site, so I have bigger fish to fry at this point. I've wasted too much time trying to chase this down. Cheers all Robi -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Monday, July 18, 2011 5:33 PM To: solr-user@lucene.apache.org Subject: Re: Analysis page output vs. actually getting search matches, a discrepency? Hmmm, is there any chance that you're stemming one place and not the other? And I infer from your output that your default search field is "moreWords", is that true and expected? You might use luke or the TermsComponent to see what's actually in the index, I'm going to guess that you'll find "sterl" but not "sterling" as an indexed term and your problem is stemming, but that's a shot in the dark. Best Erick On Mon, Jul 18, 2011 at 5:37 PM, Robert Petersen wrote: > OK I did what Hoss said, it only confirms I don't get a match when I > should and that the query parser is doing the expected. Here are the > details for one test sku. > > My analysis page output is shown in my email starting this thread and > here is my query debug output. This absolutely should match but > doesn't. Both the indexing side and the query side are splitting on > case changes. This actually isn't a problem for any of our other > content, for instance there is no issue searching for 'VideoSecu'. > Their products come up fine in our searches regardless of casing in the > query. Only SterlingTek's products seem to be causing us issues. > > Indexed content has camel case, stored in the text field 'moreWords': > "SterlingTek's NB-2LH 2 Pack Batteries + Charger Combo for Canon DC301" > Search term not matching with camel case: "SterlingTek's" > Search term matching if no case changes: "Sterlingtek's" > > Indexing: > generateWordParts="1" > generateNumberParts="1" > catenateWords="1" > catenateNumbers="1" > catenateAll="0" > splitOnCaseChange="1" > preserveOriginal="0" > /> > Searching: > generateWordParts="1" > generateNumberParts="1" > catenateWords="0" > catenateNumbers="0" > catenateAll="0" > splitOnCaseChange="1" > preserveOriginal="0" > /> > > Thanks > > http://ssdevrh01.buy.com:8983/solr/1/select?indent=on&version=2.2&q= > SterlingTek%27s&fq=&start=0&rows=1&fl=*%2Cscore&qt=standard&wt=standard& > debugQuery=on&explainOther=sku%3A216473417&hl=on&hl.fl=&echoHandler=true > &adf > > > > 0 > 4 > name="handler">org.apache.solr.handler.component.SearchHandler > > sku:216473417 > on > true > > standard > on > 1 > 2.2 > *,score > on > 0 > SterlingTek's > standard > > > > > > > SterlingTek's > SterlingTek's > PhraseQuery(moreWords:"sterling tek") > moreWords:"sterling tek" > > sku:216473417 > > > 0.0 = fieldWeight(moreWords:"sterling tek" in 76351), product of: > 0.0 = tf(phraseFreq=0.0) > 19.502613 = idf(moreWords: sterling=1 tek=72) > 0.15625 = fieldNorm(field=moreWords, doc=76351) > > > > LuceneQParser > > > > > > > > -Original Message- > From: Chris Hostetter [mailto:hossman_luc...@fucit.org] > Sent: Friday, July 15, 2011 4:36 PM > To: solr-user@lucene.apache.org > Subject: Re: Analysis page output vs. actually getting search matches, a > discrepency? > > > : Subject: Analysis page output vs. actually getting search matches, > : a discrepency? > > 9
Re: Analysis page output vs. actually getting search matches, a discrepency?
Hmmm, is there any chance that you're stemming one place and not the other? And I infer from your output that your default search field is "moreWords", is that true and expected? You might use luke or the TermsComponent to see what's actually in the index, I'm going to guess that you'll find "sterl" but not "sterling" as an indexed term and your problem is stemming, but that's a shot in the dark. Best Erick On Mon, Jul 18, 2011 at 5:37 PM, Robert Petersen wrote: > OK I did what Hoss said, it only confirms I don't get a match when I > should and that the query parser is doing the expected. Here are the > details for one test sku. > > My analysis page output is shown in my email starting this thread and > here is my query debug output. This absolutely should match but > doesn't. Both the indexing side and the query side are splitting on > case changes. This actually isn't a problem for any of our other > content, for instance there is no issue searching for 'VideoSecu'. > Their products come up fine in our searches regardless of casing in the > query. Only SterlingTek's products seem to be causing us issues. > > Indexed content has camel case, stored in the text field 'moreWords': > "SterlingTek's NB-2LH 2 Pack Batteries + Charger Combo for Canon DC301" > Search term not matching with camel case: "SterlingTek's" > Search term matching if no case changes: "Sterlingtek's" > > Indexing: > generateWordParts="1" > generateNumberParts="1" > catenateWords="1" > catenateNumbers="1" > catenateAll="0" > splitOnCaseChange="1" > preserveOriginal="0" > /> > Searching: > generateWordParts="1" > generateNumberParts="1" > catenateWords="0" > catenateNumbers="0" > catenateAll="0" > splitOnCaseChange="1" > preserveOriginal="0" > /> > > Thanks > > http://ssdevrh01.buy.com:8983/solr/1/select?indent=on&version=2.2&q= > SterlingTek%27s&fq=&start=0&rows=1&fl=*%2Cscore&qt=standard&wt=standard& > debugQuery=on&explainOther=sku%3A216473417&hl=on&hl.fl=&echoHandler=true > &adf > > > > 0 > 4 > name="handler">org.apache.solr.handler.component.SearchHandler > > sku:216473417 > on > true > > standard > on > 1 > 2.2 > *,score > on > 0 > SterlingTek's > standard > > > > > > > SterlingTek's > SterlingTek's > PhraseQuery(moreWords:"sterling tek") > moreWords:"sterling tek" > > sku:216473417 > > > 0.0 = fieldWeight(moreWords:"sterling tek" in 76351), product of: > 0.0 = tf(phraseFreq=0.0) > 19.502613 = idf(moreWords: sterling=1 tek=72) > 0.15625 = fieldNorm(field=moreWords, doc=76351) > > > > LuceneQParser > > > > > > > > -Original Message- > From: Chris Hostetter [mailto:hossman_luc...@fucit.org] > Sent: Friday, July 15, 2011 4:36 PM > To: solr-user@lucene.apache.org > Subject: Re: Analysis page output vs. actually getting search matches, a > discrepency? > > > : Subject: Analysis page output vs. actually getting search matches, > : a discrepency? > > 99% of the time when people ask questions like this, it's because of > confusion about how/when QueryParsing comes into play (as opposed to > analysis) -- analysis.jsp only shows you part of the equation, it > doesn't > know what query parser you are using. > > you mentioned that you aren't getting matches when you expect them, and > you provided the analysis.jsp output, but you didn't mention anything > about the request you are making, the query parser used etc it > owuld > be good to know the full query URL, along with the debugQuery output > showing the final query toString info. > > if that info doesn't clear up the discrepency, you should also take a > look > at the explainOther info for the doc that you expect to match that isn't > > -- if you still aren't sure what's going on, post all of that info to > solr-user and folks can probably help you make sense of it. > > (all that said: in some instances this type of problem is simply that > someone changed the schema and didn't reindex everything, so the indexed > > terms don't really match what you think they do) > > > -Hoss >
RE: Analysis page output vs. actually getting search matches, a discrepency?
OK I did what Hoss said, it only confirms I don't get a match when I should and that the query parser is doing the expected. Here are the details for one test sku. My analysis page output is shown in my email starting this thread and here is my query debug output. This absolutely should match but doesn't. Both the indexing side and the query side are splitting on case changes. This actually isn't a problem for any of our other content, for instance there is no issue searching for 'VideoSecu'. Their products come up fine in our searches regardless of casing in the query. Only SterlingTek's products seem to be causing us issues. Indexed content has camel case, stored in the text field 'moreWords': "SterlingTek's NB-2LH 2 Pack Batteries + Charger Combo for Canon DC301" Search term not matching with camel case: "SterlingTek's" Search term matching if no case changes: "Sterlingtek's" Indexing: Searching: Thanks http://ssdevrh01.buy.com:8983/solr/1/select?indent=on&version=2.2&q= SterlingTek%27s&fq=&start=0&rows=1&fl=*%2Cscore&qt=standard&wt=standard& debugQuery=on&explainOther=sku%3A216473417&hl=on&hl.fl=&echoHandler=true &adf 0 4 org.apache.solr.handler.component.SearchHandler sku:216473417 on true standard on 1 2.2 *,score on 0 SterlingTek's standard SterlingTek's SterlingTek's PhraseQuery(moreWords:"sterling tek") moreWords:"sterling tek" sku:216473417 0.0 = fieldWeight(moreWords:"sterling tek" in 76351), product of: 0.0 = tf(phraseFreq=0.0) 19.502613 = idf(moreWords: sterling=1 tek=72) 0.15625 = fieldNorm(field=moreWords, doc=76351) LuceneQParser -----Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Friday, July 15, 2011 4:36 PM To: solr-user@lucene.apache.org Subject: Re: Analysis page output vs. actually getting search matches, a discrepency? : Subject: Analysis page output vs. actually getting search matches, : a discrepency? 99% of the time when people ask questions like this, it's because of confusion about how/when QueryParsing comes into play (as opposed to analysis) -- analysis.jsp only shows you part of the equation, it doesn't know what query parser you are using. you mentioned that you aren't getting matches when you expect them, and you provided the analysis.jsp output, but you didn't mention anything about the request you are making, the query parser used etc it owuld be good to know the full query URL, along with the debugQuery output showing the final query toString info. if that info doesn't clear up the discrepency, you should also take a look at the explainOther info for the doc that you expect to match that isn't -- if you still aren't sure what's going on, post all of that info to solr-user and folks can probably help you make sense of it. (all that said: in some instances this type of problem is simply that someone changed the schema and didn't reindex everything, so the indexed terms don't really match what you think they do) -Hoss
RE: Analysis page output vs. actually getting search matches, a discrepency?
Hi Chris, Well to start from the bottom of your list there, I restrict my testing to one sku while continuously reindexing the sku after every indexer side change, and reload the core every time also. I just search from the admin page using the word in question and the exact match on the sku field (the unique one) like this: 0 6 on 0 SterlingTek's NB-2LH sku:216473417 a 10 2.2 I will have to find out more about query parsers before I can answer the rest, Will reply to that later... and it's Friday after all! :) Thanks -Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Friday, July 15, 2011 4:36 PM To: solr-user@lucene.apache.org Subject: Re: Analysis page output vs. actually getting search matches, a discrepency? : Subject: Analysis page output vs. actually getting search matches, : a discrepency? 99% of the time when people ask questions like this, it's because of confusion about how/when QueryParsing comes into play (as opposed to analysis) -- analysis.jsp only shows you part of the equation, it doesn't know what query parser you are using. you mentioned that you aren't getting matches when you expect them, and you provided the analysis.jsp output, but you didn't mention anything about the request you are making, the query parser used etc it owuld be good to know the full query URL, along with the debugQuery output showing the final query toString info. if that info doesn't clear up the discrepency, you should also take a look at the explainOther info for the doc that you expect to match that isn't -- if you still aren't sure what's going on, post all of that info to solr-user and folks can probably help you make sense of it. (all that said: in some instances this type of problem is simply that someone changed the schema and didn't reindex everything, so the indexed terms don't really match what you think they do) -Hoss
Re: Analysis page output vs. actually getting search matches, a discrepency?
: Subject: Analysis page output vs. actually getting search matches, : a discrepency? 99% of the time when people ask questions like this, it's because of confusion about how/when QueryParsing comes into play (as opposed to analysis) -- analysis.jsp only shows you part of the equation, it doesn't know what query parser you are using. you mentioned that you aren't getting matches when you expect them, and you provided the analysis.jsp output, but you didn't mention anything about the request you are making, the query parser used etc it owuld be good to know the full query URL, along with the debugQuery output showing the final query toString info. if that info doesn't clear up the discrepency, you should also take a look at the explainOther info for the doc that you expect to match that isn't -- if you still aren't sure what's going on, post all of that info to solr-user and folks can probably help you make sense of it. (all that said: in some instances this type of problem is simply that someone changed the schema and didn't reindex everything, so the indexed terms don't really match what you think they do) -Hoss
