[ http://issues.apache.org/jira/browse/HADOOP-136?page=all ]
Owen O'Malley reassigned HADOOP-136:
------------------------------------
Assignee: Hairong Kuang (was: Michel Tourn)
> Overlong UTF8's not handled well
> --------------------------------
>
> Key: HADOOP-136
> URL: http://issues.apache.org/jira/browse/HADOOP-136
> Project: Hadoop
> Issue Type: Bug
> Components: io
> Affects Versions: 0.2.0
> Reporter: Dick King
> Assigned To: Hairong Kuang
> Priority: Minor
> Fix For: 0.5.0
>
> Attachments: largeutf8.patch
>
>
> When we feed an overlong string to the UTF8 constructor, two suboptimal
> things happen.
> First, we truncate to 0xffff/3 characters on the assumption that every
> character takes three bytes in UTF8. This can truncate strings that don't
> need it, and it can be overoptimistic since there are characters that render
> as four bytes in UTF8.
> Second, the code doesn't actually handle four-byte characters.
> Third, there's a behavioral discontinuity. If the string is "discovered" to
> be overlong by the arbitrary limit described above, we truncate with a log
> message, otherwise we signal a RuntimeException. One feels that both forms
> of truncation should be treated alike. However, this issue is concealed by
> the second issue; the exception will never be thrown because UTF8.utf8Length
> can't return more than three times the length of its input.
> I would recommend changing UTF8.utf8Length to let its caller know how many
> characters of the input string will actually fit if there's an overflow
> [perhaps by returning the negative of that number] and doing the truncation
> accurately as needed.
> -dk
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
