Arawoof06 commented on code in PR #50356:
URL: https://github.com/apache/arrow/pull/50356#discussion_r3533832689
##########
cpp/src/gandiva/precompiled/string_ops.cc:
##########
@@ -208,7 +208,7 @@ gdv_int32 utf8_length_ignore_invalid(const char* data,
gdv_int32 data_len) {
}
for (int j = 1; j < char_len; ++j) {
if ((data[i + j] & 0xC0) != 0x80) { // bytes following head-byte of
glyph
- char_len += 1;
+ break;
Review Comment:
Good catch, you're right. Plain break kept char_len at the declared width so
the outer loop skipped the bytes after the mismatch. Switched to char_len = j;
break so only the confirmed bytes get consumed and the rest are re-parsed. Your
{0xF0,'a',0xE2,0x82,0xAC} case now counts 3. I added that as
TestPadMalformedUtf8KeepsValidGlyph, and the earlier {0xF0,'a','a','a'} test
now expects length 4 (lead byte + three ascii) which was the same bug hiding in
the old assertion. Verified the counts under ASAN.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
