First, your index and analysis chains are considerably different, this can easily be a source of problems. In particular, using two different tokenizers is a huge red flag. I _strongly_ recommend against this unless you’re totally sure you understand the consequences. Additionally, your use of the length filter is suspicious, especially since your problem statement is about the addition of a single letter term and the min length allowed on that filter is 2. That said, it’s reasonable to suppose that the ’a’ is filtered out in both cases, but maybe you’ve found something odd about the interactions.
Second, I have no idea what this will do. Are the equal signs typos? Used by custom code? >> https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true What does “species=“ do? That’s not Solr syntax, so it’s likely that all the params with an equal-sign are totally ignored unless it’s just a typo. Third, the easiest way to see what’s happening under the covers is to add “&debug=true” to the query and look at the parsed query. Ignore all the relevance calculations for the nonce, or specify “&debug=query” to skip that part. 90% + of the time, the question “why didn’t this query do what I expect” is answered by looking at the “&debug=query” output and the analysis page in the admin UI. NOTE: for the analysis page be sure to look at _both_ the query and index output. Also, and very important about the analysis page (and this is confusing) is that this _assumes_ that what you put in the text boxes have made it through the query parser intact and is analyzed by the field selected. Consider the search "q=field:word1 word2". Now you type “word1 word2” into the analysis text box and it looks like what you expect. That’s misleading because the query is _parsed_ as "field:word1 default_search_field:word2”. This is where “&debug=query” helps. Best, Erick > On Nov 6, 2019, at 2:36 AM, Paras Lehana <paras.leh...@indiamart.com> wrote: > > Hi Walter, > > The solr.StopFilter removes all tokens that are stopwords. Those words will >> not be in the index, so they can never match a query. > > > I think the OP's concern is different results when adding a stopword. I > think he's using the filter factory correctly - the query chain includes > the filter as well so it should remove "a" while querying. > > *@Guilherme*, please post results for both the query, the document in > result you are concerned about and post full result of analysis screen (for > both query and index). > > On Tue, 5 Nov 2019 at 21:38, Walter Underwood <wun...@wunderwood.org> wrote: > >> No. >> >> The solr.StopFilter removes all tokens that are stopwords. Those words >> will not be in the index, so they can never match a query. >> >> 1. Remove the lines with solr.StopFilter from every analysis chain in >> schema.xml. >> 2. Reload the collection, restart Solr, or whatever to read the new config. >> 3. Reindex all of the documents. >> >> When indexed with the new analysis chain, the stopwords will not be >> removed and they will be searchable. >> >> wunder >> Walter Underwood >> wun...@wunderwood.org >> http://observer.wunderwood.org/ (my blog) >> >>> On Nov 5, 2019, at 8:56 AM, Guilherme Viteri <gvit...@ebi.ac.uk> wrote: >>> >>> Ok. I am kind a lost now. >>> If I open up the console > analysis and perform it, that's the final >> result. >>> <Screenshot 2019-11-05 at 14.54.16.png> >>> >>> Your suggestion is: get rid of the <filter stopword.txt> in the >> schema.xml and during index phase replaceAll("in stopwords.txt"," ") then >> add to solr. Is that correct ? >>> >>> Thanks David >>> >>>> On 5 Nov 2019, at 14:48, David Hastings <hastings.recurs...@gmail.com >> <mailto:hastings.recurs...@gmail.com>> wrote: >>>> >>>> Fwd to another server >>>> >>>> no, >>>> <filter class="solr.StopFilterFactory" ignoreCase="true" >>>> words="stopwords.txt"/> >>>> >>>> is still using stopwords and should be removed, in my opinion of course, >>>> based on your use case may be different, but i generally axe any >> reference >>>> to them at all >>>> >>>> On Tue, Nov 5, 2019 at 9:47 AM Guilherme Viteri <gvit...@ebi.ac.uk >> <mailto:gvit...@ebi.ac.uk>> wrote: >>>> >>>>> Thanks. >>>>> Haven't I done this here ? >>>>> <fieldType name="text_field" class="solr.TextField" >>>>> positionIncrementGap="100" omitNorms="false" > >>>>> <analyzer type="index"> >>>>> <tokenizer class="solr.StandardTokenizerFactory"/> >>>>> <filter class="solr.ClassicFilterFactory"/> >>>>> <filter class="solr.LengthFilterFactory" min="2" >> max="20"/> >>>>> <filter class="solr.LowerCaseFilterFactory"/> >>>>> <filter class="solr.StopFilterFactory" ignoreCase="true" >>>>> words="stopwords.txt"/> >>>>> </analyzer> >>>>> >>>>> >>>>>> On 5 Nov 2019, at 14:15, David Hastings <hastings.recurs...@gmail.com >> <mailto:hastings.recurs...@gmail.com>> >>>>> wrote: >>>>>> >>>>>> Fwd to another server >>>>>> >>>>>> The first thing you should do is remove any reference to stop words >> and >>>>>> never use them, then re-index your data and try it again. >>>>>> >>>>>> On Tue, Nov 5, 2019 at 9:14 AM Guilherme Viteri <gvit...@ebi.ac.uk >> <mailto:gvit...@ebi.ac.uk>> >>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am performing a search to match a name (text_field), however this >> term >>>>>>> contains 'and' and 'a' and it doesn't return any records. If i remove >>>>> 'a' >>>>>>> then it works. >>>>>>> e.g >>>>>>> Search Term: lymphoid and a non-lymphoid cell >>>>>>> doesn't work: >>>>>>> >>>>> >> https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true >> < >> https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true >>> >>>>>>> < >>>>>>> >>>>> >> https://dev.reactome.org/content/query?q=lymphoid+and+a+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true >>>>>>>> >>>>>>> >>>>>>> Search term: lymphoid and non-lymphoid cell >>>>>>> works: >>>>>>> >>>>> >> https://dev.reactome.org/content/query?q=lymphoid+and+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true >>>>>>> < >>>>>>> >>>>> >> https://dev.reactome.org/content/query?q=lymphoid+and+non-lymphoid+cell&species=Homo+sapiens&species=Entries+without+species&cluster=true >>>>>>>> >>>>>>> interested in the first result >>>>>>> >>>>>>> schema.xml >>>>>>> <field name="name" type="text_field" >>>>>>> indexed="true" stored="true" omitNorms="false" required="true" >>>>>>> multiValued="false"/> >>>>>>> >>>>>>> <analyzer type="query"> >>>>>>> <tokenizer class="solr.PatternTokenizerFactory" >>>>>>> pattern="[^a-zA-Z0-9/._:]"/> >>>>>>> <filter class="solr.PatternReplaceFilterFactory" >>>>>>> pattern="^[/._:]+" replacement=""/> >>>>>>> <filter class="solr.PatternReplaceFilterFactory" >>>>>>> pattern="[/._:]+$" replacement=""/> >>>>>>> <filter class="solr.PatternReplaceFilterFactory" >>>>>>> pattern="[_]" replacement=" "/> >>>>>>> <filter class="solr.LengthFilterFactory" min="2" >>>>> max="20"/> >>>>>>> <filter class="solr.LowerCaseFilterFactory"/> >>>>>>> <filter class="solr.StopFilterFactory" >> ignoreCase="true" >>>>>>> words="stopwords.txt"/> >>>>>>> </analyzer> >>>>>>> >>>>>>> <fieldType name="text_field" class="solr.TextField" >>>>>>> positionIncrementGap="100" omitNorms="false" > >>>>>>> <analyzer type="index"> >>>>>>> <tokenizer class="solr.StandardTokenizerFactory"/> >>>>>>> <filter class="solr.ClassicFilterFactory"/> >>>>>>> <filter class="solr.LengthFilterFactory" min="2" >>>>> max="20"/> >>>>>>> <filter class="solr.LowerCaseFilterFactory"/> >>>>>>> <filter class="solr.StopFilterFactory" >> ignoreCase="true" >>>>>>> words="stopwords.txt"/> >>>>>>> </analyzer> >>>>>>> <analyzer type="query"> >>>>>>> <tokenizer class="solr.PatternTokenizerFactory" >>>>>>> pattern="[^a-zA-Z0-9/._:]"/> >>>>>>> <filter class="solr.PatternReplaceFilterFactory" >>>>>>> pattern="^[/._:]+" replacement=""/> >>>>>>> <filter class="solr.PatternReplaceFilterFactory" >>>>>>> pattern="[/._:]+$" replacement=""/> >>>>>>> <filter class="solr.PatternReplaceFilterFactory" >>>>>>> pattern="[_]" replacement=" "/> >>>>>>> <filter class="solr.LengthFilterFactory" min="2" >>>>> max="20"/> >>>>>>> <filter class="solr.LowerCaseFilterFactory"/> >>>>>>> <filter class="solr.StopFilterFactory" >> ignoreCase="true" >>>>>>> words="stopwords.txt"/> >>>>>>> </analyzer> >>>>>>> </fieldType> >>>>>>> >>>>>>> stopwords.txt >>>>>>> #Standard english stop words taken from Lucene's StopAnalyzer >>>>>>> a >>>>>>> b >>>>>>> c >>>>>>> .... >>>>>>> an >>>>>>> and >>>>>>> are >>>>>>> >>>>>>> Running SolR 6.6.2. >>>>>>> >>>>>>> Is there anything I could do to prevent this ? >>>>>>> >>>>>>> Thanks >>>>>>> Guilherme >>>>> >>>>> >>> >> >> > > -- > -- > Regards, > > *Paras Lehana* [65871] > Development Engineer, Auto-Suggest, > IndiaMART Intermesh Ltd. > > 8th Floor, Tower A, Advant-Navis Business Park, Sector 142, > Noida, UP, IN - 201303 > > Mob.: +91-9560911996 > Work: 01203916600 | Extn: *8173* > > -- > IMPORTANT: > NEVER share your IndiaMART OTP/ Password with anyone.
