Well, hypothetically, we might be able to make a reasonable attempt to
teach the filter when to strip by determine which adjacent character an
accent might be modifying and conditionally strip or not strip, but
pragmatically, this filter is used to remove Greek accents while
searching Greek texts, enabling us to perform accent-insensitive
searching of the base material. I suppose it might be useful, if a text
interspersed occasional Greek, to allow accent-insensitive searching of
any of those interspersed Greek words, but without a compelling use case
now, I'm not too inclined to improve the filter to be that much more
logical. If someone else sees a compelling need, by all means add the
extra heuristic logic. I don't consider this a bug right now. The
filter is meant to be used on UTF-8 Greek text. I certainly receive the
suggestion it might be used in another case, as you suggest, and would
consider that an improvement, if we ever have a solid use case.
Troy
On 02/21/2017 03:12 PM, DM Smith wrote:
Hypothetical: What about mixed language texts such as a Greek/French
lexicon?
DM
On Feb 21, 2017, at 4:56 PM, Troy A. Griffitts <scr...@crosswire.org
<mailto:scr...@crosswire.org>> wrote:
Simply don't use the UTF-8 Greek Accent filter on non-Greek texts. As
you have discovered there are accents used in Greek which are also
used in other languages and adverse effects will be seen for these
languages. The bottom line is simple. Only use the UTF-8 Greek
Accents filter on UTF-8 Greek texts.
Hope this helps.
On February 21, 2017 2:45:24 PM MST, David Haslam
<dfh...@googlemail.com <mailto:dfh...@googlemail.com>> wrote:
These are the principal diacritics found in Biblical Greek that have to be
removed with a UTF8GreekAccents filter.
The first five are general accents, not particular to Greek.
It's on account of these that the filter should not be applied to non-Greek
text.
U+0300 ̀ COMBINING GRAVE ACCENT
U+0301 ́ COMBINING ACUTE ACCENT
U+0308 ̈ COMBINING DIAERESIS
U+0313 ̓ COMBINING COMMA ABOVE
U+0314 ̔ COMBINING REVERSED COMMA ABOVE
U+0342 ͂ COMBINING GREEK PERISPOMENI
U+0343 ̓ COMBINING GREEK KORONIS
U+0344 ̈́ COMBINING GREEK DIALYTIKA TONOS
U+0345 ͅ COMBINING GREEK YPOGEGRAMMENI
No other diacritics or characters should be removed.
Though there are a few more combining accents in this block, they aren't
really used in Biblical Greek.
I am open to correction on this point.
e.g. The right single quotation mark (U+2019) is NOT a diacritic. It should
not be removed.
Before any of these accents can be removed, they must first be separated
from the Greek letters they are combined with.
Although normalization to the decomposed form can produce this effect, as we
have seen already, this can have undesirable side effects on any non-Greek
text in the module that may happen to include combined or unusual
characters.
It would therefore be more sensible to simply use a comprehensive mapping
table that replaces each possible accented character by the corresponding
letter in the Greek alphabet. In this way the filter can completely avoid
the need to apply any Unicode normalization.
The complete mapping table would have at least 130 rows. It will need to
take into account that there are at least 75 possible combinations of a
letter with two accents. There are none with three.
Any residual combining characters should also be removed, to cover the
possibility that a module may have been intentionally made without
normalizing the Greek source text by default to NFC.
That's my proposal. I can easily create such a mapping table that
programmers can use.
I can also readily test it with a bespoke TextPipe filter.
Best regards,
David
--
View this message in
context:http://sword-dev.350566.n4.nabble.com/GlobalOptionFilter-UTF8GreekAccents-and-non-Greek-modules-tp4656719p4656765.html
Sent from the SWORD Dev mailing list archive atNabble.com
<http://nabble.com/>.
------------------------------------------------------------------------
sword-devel mailing list:sword-devel@crosswire.org
<mailto:sword-devel@crosswire.org>
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________ sword-devel mailing
list: sword-devel@crosswire.org <mailto:sword-devel@crosswire.org>
http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to
unsubscribe/change your settings at above page
_______________________________________________
sword-devel mailing list: sword-devel@crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
_______________________________________________
sword-devel mailing list: sword-devel@crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
