Re: Stopwords in shingles suggester
I found the issue in Jira https://issues.apache.org/jira/browse/SOLR-6468 O. Klein wrote With more and more people starting to use the Suggester it seems that enablePositionIncrements for StopFilterFactory is still needed. Not sure why it is being removed from Solr5, but is there a way to keep the functionality beyond lucene 4.3 ? Or can this feature be reinstated? -- View this message in context: http://lucene.472066.n3.nabble.com/Stopwords-in-shingles-suggester-tp4166057p4186219.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Stopwords in shingles suggester
With more and more people starting to use the Suggester it seems that enablePositionIncrements for StopFilterFactory is still needed. Not sure why it is being removed from Solr5, but is there a way to keep the functionality beyond lucene 4.3 ? Or can this feature be reinstated? -- View this message in context: http://lucene.472066.n3.nabble.com/Stopwords-in-shingles-suggester-tp4166057p4185994.html Sent from the Solr - User mailing list archive at Nabble.com.
Stopwords in shingles suggester
Is there a way in Solr to filter out stopwords in shingles like ES does? http://www.elasticsearch.org/blog/searching-with-shingles/ -- View this message in context: http://lucene.472066.n3.nabble.com/Stopwords-in-shingles-suggester-tp4166057.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Stopwords in shingles suggester
You do not want stopwords in your shingles? Then put the stopword filter on top of the shingle filter. Markus -Original message- From:O. Klein kl...@octoweb.nl Sent: Monday 27th October 2014 13:56 To: solr-user@lucene.apache.org Subject: Stopwords in shingles suggester Is there a way in Solr to filter out stopwords in shingles like ES does? http://www.elasticsearch.org/blog/searching-with-shingles/ -- View this message in context: http://lucene.472066.n3.nabble.com/Stopwords-in-shingles-suggester-tp4166057.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Stopwords in shingles suggester
Configure a fieldType in schema.xml as below: fieldType name=text_shingle class=solr.TextField positionIncrementGap=0 analyzer tokenizer class=solr.StandardTokenizerFactory/ .. .. *filter class=solr.StopFilterFactory ignoreCase=true words=stopwords.txt /* filter class=solr.ShingleFilterFactory minShingleSize=2 maxShingleSize=3 outputUnigrams=false / /analyzer /fieldType Thanks, Dikshant On Mon, Oct 27, 2014 at 6:26 PM, O. Klein kl...@octoweb.nl wrote: Is there a way in Solr to filter out stopwords in shingles like ES does? http://www.elasticsearch.org/blog/searching-with-shingles/ -- View this message in context: http://lucene.472066.n3.nabble.com/Stopwords-in-shingles-suggester-tp4166057.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Stopwords in shingles suggester
On 10/27/2014 6:56 AM, O. Klein wrote: Is there a way in Solr to filter out stopwords in shingles like ES does? http://www.elasticsearch.org/blog/searching-with-shingles/ If I read that correctly, ES isn't doing anything differently than Solr does. They use the same filters that Solr does. The shingle filter doesn't have stopword support, they simply use the stopword filter (with enablePositionIncrements turned off) followed by the shingle filter. This is easy to do in schema.xml for Solr. The javadoc here shows the solr configuration for the StopFilter, with the default setting for enablePositionIncrements of true: https://lucene.apache.org/core/4_10_0/analyzers-common/org/apache/lucene/analysis/core/StopFilterFactory.html Thanks, Shawn
Re: Stopwords in shingles suggester
Is this https://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.StopFilterFactory what you are looking for? Basically, you can use analyzers for this purpose. You can even write your own analyzer. On Mon, Oct 27, 2014 at 6:26 PM, O. Klein kl...@octoweb.nl wrote: Is there a way in Solr to filter out stopwords in shingles like ES does? http://www.elasticsearch.org/blog/searching-with-shingles/ -- View this message in context: http://lucene.472066.n3.nabble.com/Stopwords-in-shingles-suggester-tp4166057.html Sent from the Solr - User mailing list archive at Nabble.com. -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax
Re: Stopwords in shingles suggester
Thank you all for your input. The stopword is being replaced by the fillerToken as shown in the article. Changing positionIncrementGap makes no difference and as of Solr 4.4, the enablePositionIncrements argument is no longer supported in the StopFilterFactory. So how do I get this working in Solr 4.5 and up? -- View this message in context: http://lucene.472066.n3.nabble.com/Stopwords-in-shingles-suggester-tp4166057p4166157.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Stopwords in shingles suggester
Hi, I think you can set fillerToken value? Ahmet On Monday, October 27, 2014 8:03 PM, O. Klein kl...@octoweb.nl wrote: Thank you all for your input. The stopword is being replaced by the fillerToken as shown in the article. Changing positionIncrementGap makes no difference and as of Solr 4.4, the enablePositionIncrements argument is no longer supported in the StopFilterFactory. So how do I get this working in Solr 4.5 and up? -- View this message in context: http://lucene.472066.n3.nabble.com/Stopwords-in-shingles-suggester-tp4166057p4166157.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Stopwords in shingles suggester
I changed luceneMatchVersion to 4.3 and got the behavior i was looking for. -- View this message in context: http://lucene.472066.n3.nabble.com/Stopwords-in-shingles-suggester-tp4166057p4166192.html Sent from the Solr - User mailing list archive at Nabble.com.
