[ http://issues.apache.org/jira/browse/HADOOP-136?page=comments#action_12378247 ]
Dick King commented on HADOOP-136: ---------------------------------- I would vote for option 2 above. I'm having difficulty imagining an existing file that has the following three properties: 1: too large to convert if need be with a one-off 2: has UTF8's larger than 32767 bytes encoded but smaller than 21845 characters 3: hasn't already been trashed by a UTF8 that was truncated. I woul dlike to go after the 4-byters as well. There's no compatibility issue here. -dk > Overlong UTF8's not handled well > -------------------------------- > > Key: HADOOP-136 > URL: http://issues.apache.org/jira/browse/HADOOP-136 > Project: Hadoop > Type: Bug > Components: io > Reporter: Dick King > Priority: Minor > > When we feed an overlong string to the UTF8 constructor, two suboptimal > things happen. > First, we truncate to 0xffff/3 characters on the assumption that every > character takes three bytes in UTF8. This can truncate strings that don't > need it, and it can be overoptimistic since there are characters that render > as four bytes in UTF8. > Second, the code doesn't actually handle four-byte characters. > Third, there's a behavioral discontinuity. If the string is "discovered" to > be overlong by the arbitrary limit described above, we truncate with a log > message, otherwise we signal a RuntimeException. One feels that both forms > of truncation should be treated alike. However, this issue is concealed by > the second issue; the exception will never be thrown because UTF8.utf8Length > can't return more than three times the length of its input. > I would recommend changing UTF8.utf8Length to let its caller know how many > characters of the input string will actually fit if there's an overflow > [perhaps by returning the negative of that number] and doing the truncation > accurately as needed. > -dk -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
