I just migrated my code from version 3.3 to the latest version 4.9.
While running my unit tests, I noticed that the feminine and masculine form
of the french words "ami" and "amie" in french did not stem to the same
token anymore.
I noticed that the french and spanish stemming algorithm had changed from
{French|Spanish}Stemmer to {French|Spanish}LightStemFilter. Code indicates
3.6
To compare both stemming algorithm (FrenchStemmer and FrenchLightStemmer),
I took a 338990 word dictionary and compared the number of resulting tokens.
FrenchStemmer : 54946 (16% of original size)
FrenchLightStemmer: 208649 (62% of original size)
I don't know the common usage of stemmers nor their performances (cpu), but
for my application, it is clear that the FrenchStemmer gives much better
stemming performances than the FrenchLightStemmer.
My question is much more out of curiosity, why is the LightStemmer the
default since 3.6?
Here are the results for the 48 words of the "abaisse" family. The stemmed
value is on the left and the indented values are words that map to the
stemmed value.
FrenchStemmer
--------------------
abaiss
abaissa
abaissable
abaissables
abaissai
abaissaient
abaissais
abaissait
abaissant
abaissante
abaissantes
abaissants
abaissas
abaissasse
abaissassent
abaissasses
abaissassiez
abaissassions
abaisse
abaisser
abaissera
abaisserai
abaisseraient
abaisserais
abaisserait
abaisseras
abaisserez
abaisseriez
abaisserions
abaisserons
abaisseront
abaisses
abaissez
abaissiez
abaissions
abaissâmes
abaissât
abaissâtes
abaissèrent
abaissé
abaissée
abaissées
abaissés
abaissement
abaissement
abaissements
abaissent
abaissent
abaisseur
abaisseur
abaisseurs
abaisson
abaissons
LightFrenchStemmer
--------------------------
abai
abaissant
abaissants
abaissement
abaissements
abais
abaisse
abaisser
abaisses
abaissé
abaissée
abaissées
abaissés
abaisa
abaissa
abaissas
abaisabl
abaissable
abaissables
abaisai
abaissai
abaissais
abaisaient
abaissaient
abaisait
abaissait
abaisam
abaissâmes
abaisant
abaissante
abaissantes
abaisas
abaissasse
abaissasses
abaisasent
abaissassent
abaisasiez
abaissassiez
abaisasion
abaissassions
abaisat
abaissât
abaissâtes
abaisent
abaissent
abaisera
abaissera
abaisseras
abaiserai
abaisserai
abaisserais
abaiseraient
abaisseraient
abaiserait
abaisserait
abaiserent
abaissèrent
abaiserez
abaisserez
abaiseriez
abaisseriez
abaiserion
abaisserions
abaiseron
abaisserons
abaiseront
abaisseront
abaiseu
abaisseur
abaisseurs
abaisez
abaissez
abaisiez
abaissiez
abaision
abaissions
abaison
abaissons
