David Mollitor created SPARK-59969:
--------------------------------------
Summary: Skip the redundant bounds check on internal UTF8String
byte reads
Key: SPARK-59969
URL: https://issues.apache.org/jira/browse/SPARK-59969
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: David Mollitor
{{UTF8String.getByte(int)}} checks that the index is in {{[0, numBytes)}} and
returns 0 otherwise. {{UTF8String}} also calls it internally from the loops
behind {{numChars}}, {{isValid}} / {{makeValid}}, {{isFullAscii}},
{{substring}}, {{charPosToByte}} / {{bytePosToChar}}, {{indexOf}}, {{reverse}},
the {{trim}} family, {{toInt}} / {{toLong}}, ASCII case conversion,
{{findInSet}}, {{soundex}} and the code point iterators. In those loops the
loop condition already keeps the index in range, so the check is redundant. The
JIT can't always remove it, especially in character walks, where each step
advances by the width of the current character.
This adds a private {{getByteUnchecked(int)}}, which asserts that the index is
in range (Spark tests run with {{-ea}}), and makes {{getByte}} call it after
its own check. The 45 internal call sites whose loop provably keeps the index
in range switch to {{getByteUnchecked}}. The other 11 stay on {{getByte}}
because they can index past the end, or depend on an invariant outside the loop:
* {{contains}}, which SPARK-59967 replaces
* {{getChar}}
* {{trimTrailingSpaces}}
* {{prevCharStart}}
* one read in {{indexOf(pattern, start, occurrence)}} that can run past the end
on truncated UTF-8
* the walks in {{levenshteinDistance}}
The behavior of the public {{getByte}} does not change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
