[
https://issues.apache.org/jira/browse/TIKA-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059894#comment-16059894
]
Tim Allison commented on TIKA-2256:
-----------------------------------
Lucene's/ICU4j's ICUFoldingFilter "normalizes" U+2F47 to U+65E5. This all
makes sense now. There's a bug in Mac's MSWord that incorrectly converts
U+65E5 to U+2F47. Preview.App must apply ICU normalization to "correct" this
back to U+65E5.
> Japanese character substituted when reading PDF
> -----------------------------------------------
>
> Key: TIKA-2256
> URL: https://issues.apache.org/jira/browse/TIKA-2256
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.14
> Reporter: Christopher Creutzig
> Attachments: mixed-fonts.pdf
>
>
> The attached file contains “日本語” in its first line. It was created on Mac OS
> X 10.11.6 by selecting “Save As PDF” in the system print dialog started from
> Microsoft Word.
> Reading the text from the PDF, the first character is not read as U+65E5, but
> as U+2F47. Copy & paste from Preview.App results in the correct U+65E5 being
> copied. (The characters look the same in some fonts, but are different.)
> The MATLAB code used for reading looks as follows:
> handler = org.apache.tika.sax.ToXMLContentHandler;
> parser = org.apache.tika.parser.AutoDetectParser;
> metadata = org.apache.tika.metadata.Metadata;
> fh = java.io.FileInputStream(fullname);
> parser.parse(fh, handler, metadata);
> s = handler.toString;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
