[
https://issues.apache.org/jira/browse/PDFBOX-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147720#comment-18147720
]
Martin Mirchev commented on PDFBOX-5126:
----------------------------------------
The surrogate-pair part of this is still present on trunk:
{{PDFTextStripper.handleDirection()}} reverses RTL runs one {{char}} at a time,
so a supplementary character in an RTL run comes out as low surrogate followed
by high surrogate. A simple case is an emoji between Hebrew words: the attached
{{rtl-emoji.pdf}} (Helvetica with a ToUnicode CMap mapping to ืฉืืื ๐ ืขืืื)
extracts as {{... 05E2 0020 DE00 D83D 0020 ...}}.
Stepping back by code point keeps the pairs in order; with this change the file
extracts as {{... 05E2 0020 D83D DE00 0020 ...}}, and LTR text and
supplementary characters in LTR runs are unchanged. It does not address the
combining-mark part of this issue.
{code}
--- a/pdfbox/src/main/java/org/apache/pdfbox/text/PDFTextStripper.java
+++ b/pdfbox/src/main/java/org/apache/pdfbox/text/PDFTextStripper.java
@@ -1949,23 +1949,19 @@
if ((level & 1) != 0)
{
- while (--end >= start)
+ while (end > start)
{
+ // step back by code point so that surrogate pairs stay in
order
+ int codePoint = word.codePointBefore(end);
+ end -= Character.charCount(codePoint);
char character = word.charAt(end);
- if (Character.isMirrored(word.codePointAt(end)))
+ if (Character.isMirrored(codePoint) &&
MIRRORING_CHAR_MAP.containsKey(character))
{
- if (MIRRORING_CHAR_MAP.containsKey(character))
- {
- result.append(MIRRORING_CHAR_MAP.get(character));
- }
- else
- {
- result.append(character);
- }
+ result.append(MIRRORING_CHAR_MAP.get(character));
}
else
{
- result.append(character);
+ result.appendCodePoint(codePoint);
}
}
}
{code}
This was found by an LLM-assisted tool and verified manually.
> Complex Unicode glyphs (surrogate pairs, combining diacritics, zero-width
> join, etc.) in a RTL context get reversed incorrectly on text extraction
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: PDFBOX-5126
> URL: https://issues.apache.org/jira/browse/PDFBOX-5126
> Project: PDFBox
> Issue Type: Bug
> Components: Text extraction
> Affects Versions: 2.0.22
> Reporter: Gรกbor Stefanik
> Priority: Major
> Attachments: rovasvegyes.pdf, rtl-emoji.pdf
>
>
> The attached PDF contains old Hungarian runic script, which is both
> right-to-left and outside Unicode's Basic Multilingual Plane (and thus
> encoded as surrogate pairs in Java's internal UTF-16-like representation).
> When this text is extracted, the surrogate pairs are reversed due to an
> overly naive use of "char"-level reversal, leading to malformed Unicode
> output.
> Likewise, when combining diacritics/modifiers occur in a right-to-left
> context, their position relative to the "parent" character is reversed, and
> so they appear on the wrong glyph, as demonstrated by the Hebrew sample in
> the same PDF. I imagine the same thing would also happen to emoji using the
> "zero-width joiner" in an RTL context.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]