but this behavior is correct, as you have position increments enabled. if you want the second query (which has 2 gaps) to match, you need to either use slop, or disable these increments alltogether.
On Fri, Apr 9, 2010 at 11:44 AM, Demian Katz <demian.k...@villanova.edu>wrote: > I've given it a try, and it definitely seems to have improved the > situation. However, there is still one weird case that's clearly related to > term positions. If I do this search, it fails: > > title:"love customs in eighteenthcentury spain" > > ...but if I do this search, it succeeds: > > title:"love customs in in eighteenthcentury spain" > > (note the duplicate "in"). > > - Demian > > > -----Original Message----- > > From: Erick Erickson [mailto:erickerick...@gmail.com] > > Sent: Thursday, April 08, 2010 11:20 AM > > To: solr-user@lucene.apache.org > > Subject: Re: solr.WordDelimiterFilterFactory problem with hyphenated > > terms? > > > > I'm not all that familiar with the underlying issues, but of the two > > I'd > > pick moving the WordDelimiterFactory rather than setting increments = > > "false". > > > > But that's at least partly a guess.... > > > > Best > > Erick > > > > On Thu, Apr 8, 2010 at 11:00 AM, Demian Katz > > <demian.k...@villanova.edu>wrote: > > > > > Thanks for looking into this -- I appreciate the help (and feel a > > little > > > better that there seems to be a bug at work here and not just my > > total > > > incomprehension). > > > > > > Sorry for any confusion over the UnicodeNormalizationFactory -- > > that's > > > actually a plug-in from the SolrMarc project ( > > > http://code.google.com/p/solrmarc/) that slipped into my example. > > Also, > > > as you guessed, my default operator is indeed set to "AND." > > > > > > It sounds to me that, of your two proposed work-arounds, moving the > > > StopFilterFactory after WordDelimiterFactory is the least disruptive. > > I'm > > > guessing that disabling position increments across the board might > > have > > > implications for other types of phrase searches, while filtering > > stopwords > > > later in the chain should be more functionally equivalent, if > > slightly less > > > efficient (potentially more terms to examine). Would you agree with > > this > > > assessment? If not, what possible negative side effects am I > > forgetting > > > about? > > > > > > thanks, > > > Demian > > > > > > > -----Original Message----- > > > > From: Erick Erickson [mailto:erickerick...@gmail.com] > > > > Sent: Wednesday, April 07, 2010 10:04 PM > > > > To: solr-user@lucene.apache.org > > > > Subject: Re: solr.WordDelimiterFilterFactory problem with > > hyphenated > > > > terms? > > > > > > > > Well, for a quick trial using trunk, I had to remove the > > > > UnicodeNormalizationFactory, is that yours? > > > > > > > > But with that removed, I get the results you do, ASSUMING that > > you've > > > > set > > > > your default operator to AND in schema.xml... > > > > > > > > Believe it or not, it all changes and all your queries return a hit > > if > > > > you > > > > do one of two things (I did this in both index and query when > > testing > > > > 'cause > > > > I'm lazy): > > > > 1> move the inclusion of the StopFilterFactory after > > > > WordDelimiterFactory > > > > or > > > > 2> for StopFilterFactory, set enablePositionIncrements="false" > > > > > > > > I think either of these might work in your situation....... > > > > > > > > On doing some more investigation, it appears that if a hyphenated > > word > > > > is > > > > immediately after a stopword AND the above is true (stop factory > > > > included > > > > before WordDelimiterFactory and enablePositionIncrements="true"), > > then > > > > the > > > > search fails. I indexed this title: > > > > > > > > Love-customs in eighteenth-century Spain for nineteenth-century > > > > > > > > Searching in solr/admin/form.jsp for: > > > > title:(nineteenth-century) > > > > > > > > fails. But if I remove the "for" from the title, the above query > > works. > > > > Searching for > > > > title:(love-customs) > > > > always works. > > > > > > > > Finally, (and it's *really* time to go to sleep now), just setting > > > > enablePositionIncrements="false" in the "index" portion of the > > schema > > > > also > > > > causes things to work. > > > > > > > > Developer folks: > > > > I didn't see anything in a quick look in SOLR or Lucene JIRAs, > > should I > > > > refine this a bit (really, sleepy time is near) and add a JIRA? > > > > > > > > Best > > > > Erick > > > > > > > > On Wed, Apr 7, 2010 at 10:29 AM, Demian Katz > > > > <demian.k...@villanova.edu>wrote: > > > > > > > > > Hello. It has been a few weeks, and I haven't gotten any > > responses. > > > > > Perhaps my question is too complicated -- maybe a better > > approach is > > > > to try > > > > > to gain enough knowledge to answer it myself. My gut feeling is > > > > still that > > > > > it's something to do with the way term positions are getting > > handled > > > > by the > > > > > WordDelimiterFilterFactory, but I don't have a good understanding > > of > > > > how > > > > > term positions are calculated or factored into searching. Can > > anyone > > > > > recommend some good reading to familiarize myself with these > > concepts > > > > in > > > > > better detail? > > > > > > > > > > thanks, > > > > > Demian > > > > > > > > > > From: Demian Katz > > > > > Sent: Tuesday, March 16, 2010 9:47 AM > > > > > To: solr-user@lucene.apache.org > > > > > Subject: solr.WordDelimiterFilterFactory problem with hyphenated > > > > terms? > > > > > > > > > > This is my first post on this list -- apologies if this has been > > > > discussed > > > > > before; I didn't come upon anything exactly equivalent in > > searching > > > > the > > > > > archives via Google. > > > > > > > > > > I'm using Solr 1.4 as part of the VuFind application, and I just > > > > noticed > > > > > that searches for hyphenated terms are failing in strange ways. > > I > > > > strongly > > > > > suspect it has something to do with the > > > > solr.WordDelimiterFilterFactory > > > > > filter, but I'm not exactly sure what. > > > > > > > > > > The problem is that I have a record with the title "Love customs > > in > > > > > eighteenth-century Spain." Depending on how I search for this, I > > get > > > > > successes or failures in a seemingly unpredictable pattern. > > > > > > > > > > Demonstration queries below were tested using the direct Solr > > > > > administration tool, just to eliminate any VuFind-related factors > > > > from the > > > > > equation while debugging. > > > > > > > > > > Queries that work: > > > > > title:(Love customs in eighteenth century Spain) > > > > > // no hyphen, no phrases > > > > > title:("Love customs in eighteenth-century Spain") > > > > > // phrase search on whole title, with hyphen > > > > > > > > > > Queries that fail: > > > > > title:(Love customs in eighteenth-century Spain) > > > > > // hyphen, no phrases > > > > > title:("Love customs in eighteenth century Spain") > > > > > // phrase search on whole title, without hyphen > > > > > title:(Love customs in "eighteenth-century" Spain) > > > > > // hyphenated word as phrase > > > > > title:(Love customs in "eighteenth century" Spain) > > > > > // hyphenated word as phrase, hyphen removed > > > > > > > > > > Here is VuFind's text field type definition: > > > > > > > > > > <fieldType name="text" class="solr.TextField" > > > > > positionIncrementGap="100"> > > > > > <analyzer type="index"> > > > > > <tokenizer class="solr.WhitespaceTokenizerFactory"/> > > > > > <filter class="solr.StopFilterFactory" ignoreCase="true" > > > > > words="stopwords.txt" enablePositionIncrements="true"/> > > > > > <filter class="solr.WordDelimiterFilterFactory" > > > > > generateWordParts="1" generateNumberParts="1" catenateWords="1" > > > > > catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> > > > > > <filter class="solr.LowerCaseFilterFactory"/> > > > > > <filter class="solr.SnowballPorterFilterFactory" > > > > language="English" > > > > > protected="protwords.txt"/> > > > > > <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> > > > > > <filter class="schema.UnicodeNormalizationFilterFactory" > > > > > version="icu4j" composed="false" remove_diacritics="true" > > > > > remove_modifiers="true" fold="true"/> > > > > > <filter class="solr.ISOLatin1AccentFilterFactory"/> > > > > > </analyzer> > > > > > <analyzer type="query"> > > > > > <tokenizer class="solr.WhitespaceTokenizerFactory"/> > > > > > <filter class="solr.SynonymFilterFactory" > > > > synonyms="synonyms.txt" > > > > > ignoreCase="true" expand="true"/> > > > > > <filter class="solr.StopFilterFactory" ignoreCase="true" > > > > > words="stopwords.txt" enablePositionIncrements="true"/> > > > > > <filter class="solr.WordDelimiterFilterFactory" > > > > > generateWordParts="1" generateNumberParts="1" catenateWords="1" > > > > > catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> > > > > > <filter class="solr.LowerCaseFilterFactory"/> > > > > > <filter class="solr.SnowballPorterFilterFactory" > > > > language="English" > > > > > protected="protwords.txt"/> > > > > > <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> > > > > > <filter class="schema.UnicodeNormalizationFilterFactory" > > > > > version="icu4j" composed="false" remove_diacritics="true" > > > > > remove_modifiers="true" fold="true"/> > > > > > <filter class="solr.ISOLatin1AccentFilterFactory"/> > > > > > </analyzer> > > > > > </fieldType> > > > > > > > > > > I did notice that in the "text" field type in VuFind's schema has > > > > > "catenateWords" and "catenateNumbers" turned on in both the index > > and > > > > query > > > > > analyzer chains. It is my understanding that these options > > should be > > > > > disabled for the query chain and only enabled for the index > > chain. > > > > However, > > > > > this may be a red herring -- I have already tried changing this > > > > setting, but > > > > > it didn't change the success/failure pattern described above. I > > have > > > > also > > > > > played with the preserveOriginal setting without apparent effect. > > > > > > > > > > From playing with the Field Analysis tool, I notice that there is > > a > > > > gap in > > > > > the term position sequence after analysis... but I'm not sure if > > > > this is > > > > > significant. > > > > > > > > > > Has anybody else run into this sort of problem? Any ideas on a > > fix? > > > > > > > > > > thanks, > > > > > Demian > > > > > > > > > > > > > > -- Robert Muir rcm...@gmail.com
