[ 
https://issues.apache.org/jira/browse/LUCENE-5057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13713487#comment-13713487
 ] 

Luca Cavanna commented on LUCENE-5057:
--------------------------------------

Thanks Adrien for looking into this, nice explanation!
                
> Hunspell stemmer generates multiple tokens
> ------------------------------------------
>
>                 Key: LUCENE-5057
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5057
>             Project: Lucene - Core
>          Issue Type: Improvement
>    Affects Versions: 4.3
>            Reporter: Luca Cavanna
>            Assignee: Adrien Grand
>
> The hunspell stemmer seems to be generating multiple tokens: the original 
> token plus the available stems.
> It might be a good thing in some cases but it seems to be a different 
> behaviour compared to the other stemmers and causes problems as well. I would 
> rather have an option to decide whether it should output only the available 
> stems, or the stems plus the original token. I'm not sure though if it's 
> possible to have only a single stem indexed, which would be even better in my 
> opinion. When I look at how snowball works only one token is indexed, the 
> stem, and that works great. Probably there's something I'm missing in how 
> hunspell works.
> Here is my issue: I have a query composed of multiple terms, which is 
> analyzed using stemming and a boolean query is generated out of it. All fine 
> when adding all clauses as should (OR operator), but if I add all clauses as 
> must (AND operator), then I can get back only the documents that contain the 
> stem originated by the exactly same original word.
> Example for the dutch language I'm working with: fiets (means bicycle in 
> dutch), its plural is fietsen.
> If I index "fietsen" I get both "fietsen" and "fiets" indexed, but if I index 
> "fiets" I get the only "fiets" indexed.
> When I query for "fietsen whatever" I get the following boolean query: 
> field:fiets field:fietsen field:whatever.
> If I apply the AND operator and use must clauses for each subquery, then I 
> can only find the documents that originally contained "fietsen", not the ones 
> that originally contained "fiets", which is not really what stemming is about.
> Any thoughts on this? I also wonder if it can be a dictionary issue since I 
> see that different words that have the word "fiets" as root don't get the 
> same stems, and using the AND operator at query time is a big issue.
> I would love to contribute on this and looking forward to your feedback.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to