Chris Karcher created LANG-977:
----------------------------------
Summary: NumericEntityEscaper incorrectly encodes supplementary
characters
Key: LANG-977
URL: https://issues.apache.org/jira/browse/LANG-977
Project: Commons Lang
Issue Type: Bug
Components: lang.text.translate.*
Affects Versions: 3.2.1
Reporter: Chris Karcher
NumericEntityEscaper will incorrectly encode supplementary unicode characters
depending on the char length of the first code point in the string.
To reproduce, run:
{code}
String escaped = NumericEntityEscaper.between(0x7f,
Integer.MAX_VALUE).translate("a \uD83D\uDC14 \uD83D\uDCA9");
{code}
Expected:
{code}
escaped == "a 🐔 💩"
{code}
Actual:
{code}
escaped == "a 🐔� 💩�
{code}
The issue lies in CharSequenceTranslator.translate() and the way it checks code
points to figure out how many characters it needs to consume. Specifically,
the issue is on [line
95|https://github.com/apache/commons-lang/blob/trunk/src/main/java/org/apache/commons/lang3/text/translate/CharSequenceTranslator.java#L95]:
{code}
// contract with translators is that they have to understand codepoints
// and they just took care of a surrogate pair
for (int pt = 0; pt < consumed; pt++) {
pos += Character.charCount(Character.codePointAt(input, pt));
}
{code}
The point of this code is to check the charCount of the character that was just
translated and move ahead by that many characters in the input string. The bug
is that it's indexing into the string using 'pt', which is _always_ 0 at the
beginning of the loop. It's effetively checking the charCount of first
character in the string every time.
A patch is attached that fixes the issue and includes supporting unit tests.
Fixing this issue in CharSequenceTranslator uncovered an issue in
CsvEscaper/CsvUnescaper caused by the fact that it wasn't respecting the "code
point contract" described in CharSequenceTranslator.translate. The fix there
was to have the translate methods return the string's code point count rather
than character count.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
