[
https://issues.apache.org/jira/browse/LUCENE-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665217#comment-13665217
]
Karl Wettin commented on LUCENE-5013:
-------------------------------------
bq. Does it make sense to have this filter do the Scandinavian folding before
the ascii folding, rather than after?
I implemented it the way I did because I want all the features of
ASCIIFoldingFilter but slightly improved for my Scandinavian corpora. I suppose
it's not completely wrong to say that ASCIIFoldingFilter is in this case used
to fold æ->ae and is thus required to be executed prior to the Scandinavian
normalization.
What possibly makes most sense it to not rely on ASCIIFoldingFilter at all. To
make it a pure ScandinavianNormalizationFilter without ü, ß and what not, that
people would have to run a second pass through some ICU-filter in order to get
that.
> ScandinavianInterintelligableASCIIFoldingFilter
> -----------------------------------------------
>
> Key: LUCENE-5013
> URL: https://issues.apache.org/jira/browse/LUCENE-5013
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/analysis
> Affects Versions: 4.3
> Reporter: Karl Wettin
> Priority: Trivial
> Attachments: LUCENE-5013.txt
>
>
> This filter is an augmentation of output from ASCIIFoldingFilter,
> it discriminate against double vowels aa, ae, ao, oe and oo, leaving just the
> first one.
> blåbærsyltetøj == blåbärsyltetöj == blaabaarsyltetoej == blabarsyltetoj
> räksmörgås == ræksmørgås == ræksmörgaos == raeksmoergaas == raksmorgas
> Caveats:
> Since this is a filtering on top of ASCIIFoldingFilter äöåøæ already has been
> folded down to aoaoae when handled by this filter it will cause effects such
> as:
> bøen -> boen -> bon
> åene -> aene -> ane
> I find this to be a trivial problem compared to not finding anything at all.
> Background:
> Swedish åäö is in fact the same letters as Norwegian and Danish åæø and thus
> interchangeable in when used between these languages. They are however folded
> differently when people type them on a keyboard lacking these characters and
> ASCIIFoldingFilter handle ä and æ differently.
> When a Swedish person is lacking umlauted characters on the keyboard they
> consistently type a, a, o instead of å, ä, ö. Foreigners also tend to use a,
> a, o.
> In Norway people tend to type aa, ae and oe instead of å, æ and ø. Some use
> a, a, o. I've also seen oo, ao, etc. And permutations. Not sure about Denmark
> but the pattern is probably the same.
> This filter solves that problem, but might also cause new.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
