[
https://issues.apache.org/jira/browse/LUCENE-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664549#comment-13664549
]
Jan Høydahl commented on LUCENE-5013:
-------------------------------------
Nice and needed component.
I have one question though, whether it is too aggressive to fold å->a, ö->o,
æ->a etc?
In my experience it is better to skip the generic folding of ø/ö->oe/o,
æ/ä->ae/a, å->aa/a which is rather destructive and instead normalize across
Norwegian/Swedish/Danish the opposite way, preserving the semantic meaning:
{noformat}
ø,ö,oe->ø
æ,ä,ae->æ
å,aa->å
{noformat}
This will support most common cases and give:
blåbærsyltetøj == blåbärsyltetöj == blaabaersyltetoej (but not blabarsyltetoj)
räksmörgås == ræksmørgås == ræksmörgaas == raeksmoergaas (but not raksmorgas)
I think this would be a good compromise which avoids many false matches between
ø/o, å/a, æ/a. One other example is the Norwegian word for "berry": *bær*. With
the aggressive approach it would be *bar* which clashes with the words for
"bare" and "bar" as well as clash with *bår* folded to *bar*. Other unfortunate
Norwegian examples are bør/bor, klær/klår/klar, får/far, før/for, klør/klor,
møte/mote, blå/bla... Perhaps the aggressive option could be a configuration
option?
Btw. I have never seen the use of eo for ø or ea for æ
bq. Though maybe the name could be simpler, (ScandinavianNormalizationFilter?)
+1
> ScandinavianInterintelligableASCIIFoldingFilter
> -----------------------------------------------
>
> Key: LUCENE-5013
> URL: https://issues.apache.org/jira/browse/LUCENE-5013
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/analysis
> Affects Versions: 4.3
> Reporter: Karl Wettin
> Priority: Trivial
> Attachments: LUCENE-5013.txt
>
>
> This filter is an augmentation of output from ASCIIFoldingFilter,
> it discriminate against double vowels aa, ae, ao, oe and oo, leaving just the
> first one.
> blåbærsyltetøj == blåbärsyltetöj == blaabaarsyltetoej == blabarsyltetoj
> räksmörgås == ræksmørgås == ræksmörgaos == raeksmoergaas == raksmorgas
> Caveats:
> Since this is a filtering on top of ASCIIFoldingFilter äöåøæ already has been
> folded down to aoaoae when handled by this filter it will cause effects such
> as:
> bøen -> boen -> bon
> åene -> aene -> ane
> I find this to be a trivial problem compared to not finding anything at all.
> Background:
> Swedish åäö is in fact the same letters as Norwegian and Danish åæø and thus
> interchangeable in when used between these languages. They are however folded
> differently when people type them on a keyboard lacking these characters and
> ASCIIFoldingFilter handle ä and æ differently.
> When a Swedish person is lacking umlauted characters on the keyboard they
> consistently type a, a, o instead of å, ä, ö. Foreigners also tend to use a,
> a, o.
> In Norway people tend to type aa, ae and oe instead of å, æ and ø. Some use
> a, a, o. I've also seen oo, ao, etc. And permutations. Not sure about Denmark
> but the pattern is probably the same.
> This filter solves that problem, but might also cause new.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
