[ https://issues.apache.org/jira/browse/JENA-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447748#comment-16447748 ]
Bruno P. Kinoshita commented on JENA-1488: ------------------------------------------ >Maybe the new filter could also be mentioned under "ConfigurableAnalyzer", >specifically under "The available TokenFilter implementations are:", where >currently ASCIIFoldingFilter is mentioned. Not kidding, but I thought about adding it there too! I didn't add it because I got confused with the GenericAnalyzer. I thought it wouldn't work because it didn't have an empty constructor (more especifically, thought it had something like this [part of the code|https://github.com/apache/jena/blob/a56f7710d88e368824042863e1ebef9afc7fd5f3/jena-text/src/main/java/org/apache/jena/query/text/assembler/GenericAnalyzerAssembler.java#L199]). My mistake. Added a mention to the new filter under ConfigurableAnalyzer. >I'm starting to think that the jena-text docs really should be split up into >several pages, they are getting hard to navigate. Code Ferret did excellent >work cleaning them up some time ago, but now the page is even longer. But >further reorganizing the docs is outside the scope of this issue. +1 ! And maybe it deserves now to have one or more entries under Tutorials as well :-) > SelectiveFoldingFilter for jena-text > ------------------------------------ > > Key: JENA-1488 > URL: https://issues.apache.org/jira/browse/JENA-1488 > Project: Apache Jena > Issue Type: Improvement > Components: Text > Affects Versions: Jena 3.6.0 > Reporter: Osma Suominen > Assignee: Bruno P. Kinoshita > Priority: Major > Fix For: Jena 3.8.0 > > > Currently there's some support for accent folding in jena-text, because > Lucene provides an ASCIIFoldingFilter. When this filter is enabled, a search > for "deja vu" will match the literal "déjà vu" in the data. > But we can't use it here at the National Library of Finland (for Finto.fi / > Skosmos), because it folds too much! In the Finnish alphabet, in addition to > the Latin a-z (which are in ASCII) we use the letters åäö and these should > not be folded to ASCII. So we need a Lucene analyzer that can be configured > with an exclude list, something like > > new SelectiveFoldingFilter(String excludeChars) > > and that can be also be configured via the Jena assembler just like other > analyzers supported by jena-text. > > This was also briefly discussed on the skosmos-users mailing list: > [https://groups.google.com/d/msg/skosmos-users/x3zR_uRBQT0/Q90-O_iDAQAJ] > Apparently Norwegians have the same problem... > I've discussed this with [~kinow] and he has some initial code to implement > this feature, so I think we can turn this into a PR fairly soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)