https://bugzilla.wikimedia.org/show_bug.cgi?id=54022
--- Comment #8 from Nemo <[email protected]> --- Sorry for going offtopic with my stupid questions, mainly I'd like to make a list of possible weaknesses e.g. for Italian analysis so that users can specifically test them a bit. (In reply to comment #7) > Right, this only effects English. > > Unfortunately (or fortunately for a small set of use cases) there aren't as > many different options for languages other than English. I believe we have > five options, in order of how much they increase recall and decrease > precision: > 1. No stemming > 2. Minimal (just possessives) > 3. KStem > 4. Porter Stemmer > 5. Porter Stemmer via Snowball > > A few other languages have "minimal" (or "light") stemmers in addition to > their > more aggressive versions. In all cases other than English at this point we > use > the Elasticsearch default which is the more aggressive version. Our default is standard i.e. http://www.elasticsearch.org/guide/reference/index-modules/analysis/standard-tokenizer/ or the language default for those which have one ( http://www.elasticsearch.org/guide/reference/index-modules/analysis/lang-analyzer/ ) so the stopwords we're using are those linked from http://www.elasticsearch.org/guide/reference/index-modules/analysis/snowball-analyzer/ ? > > Switching from the Elasticsearch default to a customized version isn't hard > and > we're totally willing to do it. Good! I guess you'll need help from native speakers and that they'll need some pointers from the docs on how to help. 30 languages < 285, so maybe – when you start expanding to many languages – as a starting point cutoff_frequency can be used to replace stopwords lists where one is not available as mentioned in https://gibrown.wordpress.com/2013/05/01/three-principles-for-multilingal-indexing-in-elasticsearch/ ? That would be a possible enhancement to file separately. -- You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ Wikibugs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
