Re: Problem with caps and star symbol
Its Working as I was looking for.Thanks Mr. Erick. On Wed, Jun 1, 2011 at 8:29 PM, Erick Erickson wrote: > Take a look here: > > http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.WordDelimiterFilterFactory > > I think you want generateWordParts=1, catenateWords=1 and > preserveOriginal=1, > but check it out with the admin/analysis page. > > Oh, and your index-time and query-time patterns for WDFF will probably > be different, see > the example schema. > > Best > Erick > > On Wed, Jun 1, 2011 at 7:40 AM, Saumitra Chowdhury > wrote: > > Thanks for your point. I was really tripping that issue. But Now I need a > > bit help more. > > As far I have noticed that in the case of a value like "*role_delete*" , > > WordDelimiterFilterFactory > > index two words like "*role"* and "*delete"* and in both search result > with > > the term "*role*" and "*delete*" will > > include that document. > > > > Now In the case of the value like "*role_delete*" I want to index all > four > > terms like [ *role_delete, roledelete, role, delete ].* > > In total both the original and processed word by > WordDelimiterFilterFactory > > will be indexed. > > > > Is it possible ?? Does any additional filter with > WordDelimiterFilterFactory > > can do that ?? Or > > any filter can do such like operation ?? > > > > On Tue, May 31, 2011 at 8:07 PM, Erick Erickson >wrote: > > > >> I think you're tripping over the issue that wildcards aren't analyzed, > they > >> don't go through your analysis chain. So the casing matters. Try > >> lowercasing > >> the input and I believe you'll see more like what you expect... > >> > >> Best > >> Erick > >> > >> On Mon, May 30, 2011 at 12:07 AM, Saumitra Chowdhury > >> wrote: > >> > I am sending some xml to understand the scenario. > >> > Indexed term = ROLE_DELETE > >> > Search Term = roledelete > >> > > >> > > >> > 0 > >> > 4 > >> > > >> > on > >> > 0 > >> > name : roledelete > >> > 2.2 > >> > 10 > >> > > >> > > >> > > >> > > >> > Indexed term = ROLE_DELETE > >> > Search Term = role > >> > > >> > > >> > 0 > >> > 5 > >> > > >> > on > >> > 0 > >> > name : role > >> > 2.2 > >> > 10 > >> > > >> > > >> > > >> > > >> > Mon May 30 13:09:14 BDST 2011 > >> > Global Role for Deletion > >> > role:9223372036854775802 > >> > Mon May 30 13:09:14 BDST 2011 > >> > ROLE_DELETE > >> > > >> > > >> > > >> > > >> > Mon May 30 13:09:14 BDST 2011 > >> > Global Role for Deletion > >> > role:9223372036854775802 > >> > Mon May 30 13:09:14 BDST 2011 > >> > ROLE_DELETE > >> > > >> > > >> > > >> > > >> > > >> > Indexed term = ROLE_DELETE > >> > Search Term = role* > >> > > >> > > >> > 0 > >> > 4 > >> > > >> > on > >> > 0 > >> > name : role* > >> > 2.2 > >> > 10 > >> > > >> > > >> > > >> > > >> > Mon May 30 13:09:14 BDST 2011 > >> > Global Role for Deletion > >> > role:9223372036854775802 > >> > Mon May 30 13:09:14 BDST 2011 > >> > ROLE_DELETE > >> > > >> > > >> > > >> > > >> > > >> > Indexed term = ROLE_DELETE > >> > Search Term = Role* > >> > > >> > > >> > 0 > >> > 4 > >> > > >> > on > >> > 0 > >> > name : Role* > >> > 2.2 > >> > 10 > >> > > >> > > >> > > >> > > >> > > >> > > >> > Indexed term = ROLE_DELETE > >> > Search Term = ROLE_DELETE* > >> > > >> > > >> > 0 > >> > 4 > >> > > >> > on > >> > 0 > >> > name : ROLE_DELETE* > >> > 2.2 > >> > 10 > >> > > >> > > >> > > >> > > >> > I am also adding a analysis html. > >> > > >> > > >> > On Mon, May 30, 2011 at 7:19 AM, Erick Erickson < > erickerick...@gmail.com > >> > > >> > wrote: > >> >> > >> >> I'd start by looking at the analysis page from the Solr admin page. > That > >> >> will give you an idea of the transformations the various steps carry > >> out, > >> >> it's invaluable! > >> >> > >> >> Best > >> >> Erick > >> >> On May 26, 2011 12:53 AM, "Saumitra Chowdhury" < > >> >> saumi...@smartitengineering.com> wrote: > >> >> > Hi all , > >> >> > In my schema.xml i am using WordDelimiterFilterFactory, > >> >> > LowerCaseFilterFactory, StopFilterFactory for index analyzer and an > >> >> > extra > >> >> > SynonymFilterFactory for query analyzer. I am indexing a field name > >> >> > '*name*'.Now > >> >> > if a value with all caps like "NAME_BILL" is indexed I am able get > >> this > >> >> > as > >> >> > search result with the term " *name_bill *", " *NAME_BILL *", " > >> >> > *namebill > >> >> *", > >> >> > "*namebill** ", " *nameb** " ... But for the term like following " > * > >> >> > NAME_BILL** ", " *name_bill** ", " *namebill** ", " *NAME** " the > >> result > >> >> > does mot show this document. Can anyone please explain why this is > >> >> > happening? .In fact star " * " is not giving any result in many > >> >> > cases specially if it is used after full value of a field. > >> >> > > >> >> > Portion of my schema is given below. > >> >> > > >> >> > >> >> positionIncrementGap="100"> > >> >> > - > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > - > >> >> > >> >> > positionIncrementGap="100"> > >> >> >
Re: Problem with caps and star symbol
Take a look here: http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.WordDelimiterFilterFactory I think you want generateWordParts=1, catenateWords=1 and preserveOriginal=1, but check it out with the admin/analysis page. Oh, and your index-time and query-time patterns for WDFF will probably be different, see the example schema. Best Erick On Wed, Jun 1, 2011 at 7:40 AM, Saumitra Chowdhury wrote: > Thanks for your point. I was really tripping that issue. But Now I need a > bit help more. > As far I have noticed that in the case of a value like "*role_delete*" , > WordDelimiterFilterFactory > index two words like "*role"* and "*delete"* and in both search result with > the term "*role*" and "*delete*" will > include that document. > > Now In the case of the value like "*role_delete*" I want to index all four > terms like [ *role_delete, roledelete, role, delete ].* > In total both the original and processed word by WordDelimiterFilterFactory > will be indexed. > > Is it possible ?? Does any additional filter with WordDelimiterFilterFactory > can do that ?? Or > any filter can do such like operation ?? > > On Tue, May 31, 2011 at 8:07 PM, Erick Erickson > wrote: > >> I think you're tripping over the issue that wildcards aren't analyzed, they >> don't go through your analysis chain. So the casing matters. Try >> lowercasing >> the input and I believe you'll see more like what you expect... >> >> Best >> Erick >> >> On Mon, May 30, 2011 at 12:07 AM, Saumitra Chowdhury >> wrote: >> > I am sending some xml to understand the scenario. >> > Indexed term = ROLE_DELETE >> > Search Term = roledelete >> > >> > >> > 0 >> > 4 >> > >> > on >> > 0 >> > name : roledelete >> > 2.2 >> > 10 >> > >> > >> > >> > >> > Indexed term = ROLE_DELETE >> > Search Term = role >> > >> > >> > 0 >> > 5 >> > >> > on >> > 0 >> > name : role >> > 2.2 >> > 10 >> > >> > >> > >> > >> > Mon May 30 13:09:14 BDST 2011 >> > Global Role for Deletion >> > role:9223372036854775802 >> > Mon May 30 13:09:14 BDST 2011 >> > ROLE_DELETE >> > >> > >> > >> > >> > Mon May 30 13:09:14 BDST 2011 >> > Global Role for Deletion >> > role:9223372036854775802 >> > Mon May 30 13:09:14 BDST 2011 >> > ROLE_DELETE >> > >> > >> > >> > >> > >> > Indexed term = ROLE_DELETE >> > Search Term = role* >> > >> > >> > 0 >> > 4 >> > >> > on >> > 0 >> > name : role* >> > 2.2 >> > 10 >> > >> > >> > >> > >> > Mon May 30 13:09:14 BDST 2011 >> > Global Role for Deletion >> > role:9223372036854775802 >> > Mon May 30 13:09:14 BDST 2011 >> > ROLE_DELETE >> > >> > >> > >> > >> > >> > Indexed term = ROLE_DELETE >> > Search Term = Role* >> > >> > >> > 0 >> > 4 >> > >> > on >> > 0 >> > name : Role* >> > 2.2 >> > 10 >> > >> > >> > >> > >> > >> > >> > Indexed term = ROLE_DELETE >> > Search Term = ROLE_DELETE* >> > >> > >> > 0 >> > 4 >> > >> > on >> > 0 >> > name : ROLE_DELETE* >> > 2.2 >> > 10 >> > >> > >> > >> > >> > I am also adding a analysis html. >> > >> > >> > On Mon, May 30, 2011 at 7:19 AM, Erick Erickson > > >> > wrote: >> >> >> >> I'd start by looking at the analysis page from the Solr admin page. That >> >> will give you an idea of the transformations the various steps carry >> out, >> >> it's invaluable! >> >> >> >> Best >> >> Erick >> >> On May 26, 2011 12:53 AM, "Saumitra Chowdhury" < >> >> saumi...@smartitengineering.com> wrote: >> >> > Hi all , >> >> > In my schema.xml i am using WordDelimiterFilterFactory, >> >> > LowerCaseFilterFactory, StopFilterFactory for index analyzer and an >> >> > extra >> >> > SynonymFilterFactory for query analyzer. I am indexing a field name >> >> > '*name*'.Now >> >> > if a value with all caps like "NAME_BILL" is indexed I am able get >> this >> >> > as >> >> > search result with the term " *name_bill *", " *NAME_BILL *", " >> >> > *namebill >> >> *", >> >> > "*namebill** ", " *nameb** " ... But for the term like following " * >> >> > NAME_BILL** ", " *name_bill** ", " *namebill** ", " *NAME** " the >> result >> >> > does mot show this document. Can anyone please explain why this is >> >> > happening? .In fact star " * " is not giving any result in many >> >> > cases specially if it is used after full value of a field. >> >> > >> >> > Portion of my schema is given below. >> >> > >> >> > > >> positionIncrementGap="100"> >> >> > - >> >> > >> >> > >> >> > >> >> > >> >> > - >> >> > > >> > positionIncrementGap="100"> >> >> > - >> >> > >> >> > >> >> > > >> > generateNumberParts="0" catenateWords="1" catenateNumbers="1" >> >> > catenateAll="0"/> >> >> > >> >> > > >> > words="stopwords.txt" enablePositionIncrements="true"/> >> >> > >> >> > - >> >> > >> >> > >> >> > > >> > generateNumberParts="0" catenateWords="1" catenateNumbers="1" >> >> > catenateAll="0"/> >> >> > >> >> > > >> > ignoreCase="true" expand="true"/> >> >> > > >> > words="stopwords.txt" enablePositionIncrements="true"/> >> >> > >> >> > >> >> > - >> >> > > >> > positionIncrementGap="100"> >> >> > - >>
Re: Problem with caps and star symbol
Thanks for your point. I was really tripping that issue. But Now I need a bit help more. As far I have noticed that in the case of a value like "*role_delete*" , WordDelimiterFilterFactory index two words like "*role"* and "*delete"* and in both search result with the term "*role*" and "*delete*" will include that document. Now In the case of the value like "*role_delete*" I want to index all four terms like [ *role_delete, roledelete, role, delete ].* In total both the original and processed word by WordDelimiterFilterFactory will be indexed. Is it possible ?? Does any additional filter with WordDelimiterFilterFactory can do that ?? Or any filter can do such like operation ?? On Tue, May 31, 2011 at 8:07 PM, Erick Erickson wrote: > I think you're tripping over the issue that wildcards aren't analyzed, they > don't go through your analysis chain. So the casing matters. Try > lowercasing > the input and I believe you'll see more like what you expect... > > Best > Erick > > On Mon, May 30, 2011 at 12:07 AM, Saumitra Chowdhury > wrote: > > I am sending some xml to understand the scenario. > > Indexed term = ROLE_DELETE > > Search Term = roledelete > > > > > > 0 > > 4 > > > > on > > 0 > > name : roledelete > > 2.2 > > 10 > > > > > > > > > > Indexed term = ROLE_DELETE > > Search Term = role > > > > > > 0 > > 5 > > > > on > > 0 > > name : role > > 2.2 > > 10 > > > > > > > > > > Mon May 30 13:09:14 BDST 2011 > > Global Role for Deletion > > role:9223372036854775802 > > Mon May 30 13:09:14 BDST 2011 > > ROLE_DELETE > > > > > > > > > > Mon May 30 13:09:14 BDST 2011 > > Global Role for Deletion > > role:9223372036854775802 > > Mon May 30 13:09:14 BDST 2011 > > ROLE_DELETE > > > > > > > > > > > > Indexed term = ROLE_DELETE > > Search Term = role* > > > > > > 0 > > 4 > > > > on > > 0 > > name : role* > > 2.2 > > 10 > > > > > > > > > > Mon May 30 13:09:14 BDST 2011 > > Global Role for Deletion > > role:9223372036854775802 > > Mon May 30 13:09:14 BDST 2011 > > ROLE_DELETE > > > > > > > > > > > > Indexed term = ROLE_DELETE > > Search Term = Role* > > > > > > 0 > > 4 > > > > on > > 0 > > name : Role* > > 2.2 > > 10 > > > > > > > > > > > > > > Indexed term = ROLE_DELETE > > Search Term = ROLE_DELETE* > > > > > > 0 > > 4 > > > > on > > 0 > > name : ROLE_DELETE* > > 2.2 > > 10 > > > > > > > > > > I am also adding a analysis html. > > > > > > On Mon, May 30, 2011 at 7:19 AM, Erick Erickson > > > wrote: > >> > >> I'd start by looking at the analysis page from the Solr admin page. That > >> will give you an idea of the transformations the various steps carry > out, > >> it's invaluable! > >> > >> Best > >> Erick > >> On May 26, 2011 12:53 AM, "Saumitra Chowdhury" < > >> saumi...@smartitengineering.com> wrote: > >> > Hi all , > >> > In my schema.xml i am using WordDelimiterFilterFactory, > >> > LowerCaseFilterFactory, StopFilterFactory for index analyzer and an > >> > extra > >> > SynonymFilterFactory for query analyzer. I am indexing a field name > >> > '*name*'.Now > >> > if a value with all caps like "NAME_BILL" is indexed I am able get > this > >> > as > >> > search result with the term " *name_bill *", " *NAME_BILL *", " > >> > *namebill > >> *", > >> > "*namebill** ", " *nameb** " ... But for the term like following " * > >> > NAME_BILL** ", " *name_bill** ", " *namebill** ", " *NAME** " the > result > >> > does mot show this document. Can anyone please explain why this is > >> > happening? .In fact star " * " is not giving any result in many > >> > cases specially if it is used after full value of a field. > >> > > >> > Portion of my schema is given below. > >> > > >> > >> positionIncrementGap="100"> > >> > - > >> > > >> > > >> > > >> > > >> > - > >> > >> > positionIncrementGap="100"> > >> > - > >> > > >> > > >> > >> > generateNumberParts="0" catenateWords="1" catenateNumbers="1" > >> > catenateAll="0"/> > >> > > >> > >> > words="stopwords.txt" enablePositionIncrements="true"/> > >> > > >> > - > >> > > >> > > >> > >> > generateNumberParts="0" catenateWords="1" catenateNumbers="1" > >> > catenateAll="0"/> > >> > > >> > >> > ignoreCase="true" expand="true"/> > >> > >> > words="stopwords.txt" enablePositionIncrements="true"/> > >> > > >> > > >> > - > >> > >> > positionIncrementGap="100"> > >> > - > >> > > >> > > >> > >> > generateNumberParts="0" catenateWords="1" catenateNumbers="1" > >> > catenateAll="0"/> > >> > > >> > >> > ignoreCase="true" expand="false"/> > >> > >> > words="stopwords.txt"/> > >> > > >> > > >> > > > > > >
Re: Problem with caps and star symbol
I think you're tripping over the issue that wildcards aren't analyzed, they don't go through your analysis chain. So the casing matters. Try lowercasing the input and I believe you'll see more like what you expect... Best Erick On Mon, May 30, 2011 at 12:07 AM, Saumitra Chowdhury wrote: > I am sending some xml to understand the scenario. > Indexed term = ROLE_DELETE > Search Term = roledelete > > > 0 > 4 > > on > 0 > name : roledelete > 2.2 > 10 > > > > > Indexed term = ROLE_DELETE > Search Term = role > > > 0 > 5 > > on > 0 > name : role > 2.2 > 10 > > > > > Mon May 30 13:09:14 BDST 2011 > Global Role for Deletion > role:9223372036854775802 > Mon May 30 13:09:14 BDST 2011 > ROLE_DELETE > > > > > Mon May 30 13:09:14 BDST 2011 > Global Role for Deletion > role:9223372036854775802 > Mon May 30 13:09:14 BDST 2011 > ROLE_DELETE > > > > > > Indexed term = ROLE_DELETE > Search Term = role* > > > 0 > 4 > > on > 0 > name : role* > 2.2 > 10 > > > > > Mon May 30 13:09:14 BDST 2011 > Global Role for Deletion > role:9223372036854775802 > Mon May 30 13:09:14 BDST 2011 > ROLE_DELETE > > > > > > Indexed term = ROLE_DELETE > Search Term = Role* > > > 0 > 4 > > on > 0 > name : Role* > 2.2 > 10 > > > > > > > Indexed term = ROLE_DELETE > Search Term = ROLE_DELETE* > > > 0 > 4 > > on > 0 > name : ROLE_DELETE* > 2.2 > 10 > > > > > I am also adding a analysis html. > > > On Mon, May 30, 2011 at 7:19 AM, Erick Erickson > wrote: >> >> I'd start by looking at the analysis page from the Solr admin page. That >> will give you an idea of the transformations the various steps carry out, >> it's invaluable! >> >> Best >> Erick >> On May 26, 2011 12:53 AM, "Saumitra Chowdhury" < >> saumi...@smartitengineering.com> wrote: >> > Hi all , >> > In my schema.xml i am using WordDelimiterFilterFactory, >> > LowerCaseFilterFactory, StopFilterFactory for index analyzer and an >> > extra >> > SynonymFilterFactory for query analyzer. I am indexing a field name >> > '*name*'.Now >> > if a value with all caps like "NAME_BILL" is indexed I am able get this >> > as >> > search result with the term " *name_bill *", " *NAME_BILL *", " >> > *namebill >> *", >> > "*namebill** ", " *nameb** " ... But for the term like following " * >> > NAME_BILL** ", " *name_bill** ", " *namebill** ", " *NAME** " the result >> > does mot show this document. Can anyone please explain why this is >> > happening? .In fact star " * " is not giving any result in many >> > cases specially if it is used after full value of a field. >> > >> > Portion of my schema is given below. >> > >> > > positionIncrementGap="100"> >> > - >> > >> > >> > >> > >> > - >> > > > positionIncrementGap="100"> >> > - >> > >> > >> > > > generateNumberParts="0" catenateWords="1" catenateNumbers="1" >> > catenateAll="0"/> >> > >> > > > words="stopwords.txt" enablePositionIncrements="true"/> >> > >> > - >> > >> > >> > > > generateNumberParts="0" catenateWords="1" catenateNumbers="1" >> > catenateAll="0"/> >> > >> > > > ignoreCase="true" expand="true"/> >> > > > words="stopwords.txt" enablePositionIncrements="true"/> >> > >> > >> > - >> > > > positionIncrementGap="100"> >> > - >> > >> > >> > > > generateNumberParts="0" catenateWords="1" catenateNumbers="1" >> > catenateAll="0"/> >> > >> > > > ignoreCase="true" expand="false"/> >> > > > words="stopwords.txt"/> >> > >> > >> > > >
Re: Problem with caps and star symbol
I am sending some xml to understand the scenario. Indexed term = ROLE_DELETE Search Term = roledelete 0 4 on 0 name : roledelete 2.2 10 Indexed term = ROLE_DELETE Search Term = role 0 5 on 0 name : role 2.2 10 Mon May 30 13:09:14 BDST 2011 Global Role for Deletion role:9223372036854775802 Mon May 30 13:09:14 BDST 2011 ROLE_DELETE Mon May 30 13:09:14 BDST 2011 Global Role for Deletion role:9223372036854775802 Mon May 30 13:09:14 BDST 2011 ROLE_DELETE Indexed term = ROLE_DELETE Search Term = role* 0 4 on 0 name : role* 2.2 10 Mon May 30 13:09:14 BDST 2011 Global Role for Deletion role:9223372036854775802 Mon May 30 13:09:14 BDST 2011 ROLE_DELETE Indexed term = ROLE_DELETE Search Term = Role* 0 4 on 0 name : Role* 2.2 10 Indexed term = ROLE_DELETE Search Term = ROLE_DELETE* 0 4 on 0 name : ROLE_DELETE* 2.2 10 I am also adding a analysis html. On Mon, May 30, 2011 at 7:19 AM, Erick Erickson wrote: > I'd start by looking at the analysis page from the Solr admin page. That > will give you an idea of the transformations the various steps carry out, > it's invaluable! > > Best > Erick > On May 26, 2011 12:53 AM, "Saumitra Chowdhury" < > saumi...@smartitengineering.com> wrote: > > Hi all , > > In my schema.xml i am using WordDelimiterFilterFactory, > > LowerCaseFilterFactory, StopFilterFactory for index analyzer and an extra > > SynonymFilterFactory for query analyzer. I am indexing a field name > > '*name*'.Now > > if a value with all caps like "NAME_BILL" is indexed I am able get this > as > > search result with the term " *name_bill *", " *NAME_BILL *", " *namebill > *", > > "*namebill** ", " *nameb** " ... But for the term like following " * > > NAME_BILL** ", " *name_bill** ", " *namebill** ", " *NAME** " the result > > does mot show this document. Can anyone please explain why this is > > happening? .In fact star " * " is not giving any result in many > > cases specially if it is used after full value of a field. > > > > Portion of my schema is given below. > > > > positionIncrementGap="100"> > > - > > > > > > > > > > - > > > > - > > > > > > > generateNumberParts="0" catenateWords="1" catenateNumbers="1" > > catenateAll="0"/> > > > > > words="stopwords.txt" enablePositionIncrements="true"/> > > > > - > > > > > > > generateNumberParts="0" catenateWords="1" catenateNumbers="1" > > catenateAll="0"/> > > > > > ignoreCase="true" expand="true"/> > > > words="stopwords.txt" enablePositionIncrements="true"/> > > > > > > - > > > positionIncrementGap="100"> > > - > > > > > > > generateNumberParts="0" catenateWords="1" catenateNumbers="1" > > catenateAll="0"/> > > > > > ignoreCase="true" expand="false"/> > > > words="stopwords.txt"/> > > > > > > >
Re: Problem with caps and star symbol
I'd start by looking at the analysis page from the Solr admin page. That will give you an idea of the transformations the various steps carry out, it's invaluable! Best Erick On May 26, 2011 12:53 AM, "Saumitra Chowdhury" < saumi...@smartitengineering.com> wrote: > Hi all , > In my schema.xml i am using WordDelimiterFilterFactory, > LowerCaseFilterFactory, StopFilterFactory for index analyzer and an extra > SynonymFilterFactory for query analyzer. I am indexing a field name > '*name*'.Now > if a value with all caps like "NAME_BILL" is indexed I am able get this as > search result with the term " *name_bill *", " *NAME_BILL *", " *namebill *", > "*namebill** ", " *nameb** " ... But for the term like following " * > NAME_BILL** ", " *name_bill** ", " *namebill** ", " *NAME** " the result > does mot show this document. Can anyone please explain why this is > happening? .In fact star " * " is not giving any result in many > cases specially if it is used after full value of a field. > > Portion of my schema is given below. > > > - > > > > > - > > - > > > generateNumberParts="0" catenateWords="1" catenateNumbers="1" > catenateAll="0"/> > > words="stopwords.txt" enablePositionIncrements="true"/> > > - > > > generateNumberParts="0" catenateWords="1" catenateNumbers="1" > catenateAll="0"/> > > ignoreCase="true" expand="true"/> > words="stopwords.txt" enablePositionIncrements="true"/> > > > - > positionIncrementGap="100"> > - > > > generateNumberParts="0" catenateWords="1" catenateNumbers="1" > catenateAll="0"/> > > ignoreCase="true" expand="false"/> > words="stopwords.txt"/> > > >
