You might also take a machine learning approach, like Naïve Bayesian 
Classification. For instance
 You build test sets from records in various scripts and use the classifier to 
find hybrid cases.  I have quite satisfactory results with this approach in a 
slightly different use case.

From: George Milten []
Sent: dinsdag 10 februari 2015 16:09
To: Kool,Wouter
Subject: Re: UNICODE character identification

yes probably this is where i was also heading, but thought there was a more 
clever way. Also, is there a good perl normaliser? I have not had any 
experience with:

For starters if i could spot only the odd letters between latin and greek regex 
character classes, i would be more than happy

2015-02-10 17:04 GMT+02:00 Kool,Wouter 
Apologies, I missed the subject line...

Then you might use the regex character classes. For instance $text =~ 
m/\p{Hiragana}/;  matches any Japanese Hiragana character. I have not tested 
it, but I suppose /[^\p{Latin}]/ would match any non-latin characters. So you 
find the character class that most characters match and you look for the 
exceptions. Would that help?

From: George Milten 
Sent: dinsdag 10 februari 2015 15:56
To: Kool,Wouter
Subject: Re: UNICODE character identification


thank you

2015-02-10 16:54 GMT+02:00 Kool,Wouter 
What encoding is your data in? utf8? Single-byte encoding? Marc8? That 
information matters a lot to determine whether your idea would work. If it is 
in a single-byte encoding there is often no way to determine the script the 
character belongs to.

Wouter Kool
Metadata Specialist · OCLC B.V.
Schipholweg 99 · P.O. Box 876 · 2300 AW Leiden · The Netherlands
t +31-(0)71-524 6500<tel:%2B31-%280%2971-524%206500><> ·<>
[Volg @OCLC_NL op Twitter]<>[Volg OCLC (Nederland) 
op LinkedIn]<>[Abonneer op 

From: George Milten 
Sent: dinsdag 10 februari 2015 13:27
Subject: UNICODE character identification

Hello friendly folks,

follows what i am trying to do, and i am looking for your help in order to find 
the most clever way to achieve this:

We have records, that include typos like this: we have a word say Plato, where 
the last o is inputted with the keyboard set to Greek language, so we need 
something that would parse all metadata in a per character basis, check against 
what is the script language that the majority of characters the word belongs to 
have, and return the odd characters, the script they belong, and the record 
identifier they were found in, so as to be able to correct them

thank you in advance

Reply via email to