Hi Peter,
I can think of a few ways to do this. One idea is to use a thesaurus and just
add all of the terms to the thesaurus as expansions of the terms with the funny
characters. It might be hard to know all your terms before the search.
Another way is to just parse your search string for the offending characters
and change the search to an or-query of the original term and the term with the
replaced character. I think this should work OK as long as there are not a
huge number of replaced characters, and as long as the search strings are not
very large. Here is a hacky example of what I mean--if you have a search
parser already, something like this would be relatively easy to add I think.
let $search := "Jacob Ørn"
let $searchTokens := fn:tokenize($search, " ")
let $replacedTokens :=
for $token in $searchTokens
return if ( fn:contains($token, "Ø") )
then (fn:replace($token, "Ø", "O") )
else ()
return
cts:or-query((
cts:and-query((
for $tok in $searchTokens return cts:word-query($tok) )),
cts:or-query((
for $orTok in $replacedTokens return cts:word-query($orTok) ))
))
I am sure there are other ways as well. Hope this helps.
-Danny
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Hickman
Sent: Monday, July 21, 2008 6:52 AM
To: MarkLogic ML
Subject: [MarkLogic Dev General] Unicode flattening for non combinedcharacters.
Our client has data such as "Jacob Ørn" that they want to search for.
The are expecting that searching for "orn" would match "Ørn" as they see
"Ø" as an accented character. According to the Unicode Standard 4.0
(always a good read :)) U+00D8 "Latin Capital Letter O With Stroke" is
not a combined character and therefore is not matched by "O" when doing
a case and diacritical insensitive search. This is what I expect and
understand as a developer.
Is there some way of getting client's expected behaviour? I suspect that
the "Ø" is only one of several characters that have this problem, such
as the "Ł" (U+0141) in "Łodz".
--
Peter Hickman.
Semantico, Lees House, 21-23 Dyke Road, Brighton BN1 3FE
t: 01273 358223
f: 01273 723232
e: [EMAIL PROTECTED]
w: www.semantico.com
_______________________________________________
General mailing list
[email protected]
http://xqzone.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected]
http://xqzone.com/mailman/listinfo/general
