David Mollitor created SPARK-59952:
--------------------------------------
Summary: Avoid allocating lowercased copies in UTF8_LCASE contains
for ASCII strings
Key: SPARK-59952
URL: https://issues.apache.org/jira/browse/SPARK-59952
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: David Mollitor
h3. What changes were proposed in this pull request?
For the UTF8_LCASE collation, {{CollationSupport.Contains}} calls
{{{}CollationAwareUTF8String.lowercaseContains{}}}. When both operands are full
ASCII, its fast path ran
{{{}target.toLowerCase().contains(pattern.toLowerCase()){}}}, which builds a
lowercased copy of the whole target (a new byte array plus a new
{{{}UTF8String{}}}) and of the pattern, searches the copies, and then throws
them away.
This PR replaces that fast path with {{{}lowercaseContainsAscii{}}}, which
searches the original bytes and folds case as it compares, so it allocates
nothing:
* ASCII case folding maps each byte to one byte ({{{}A{}}}\-{{Z}} to
{{a}}-{{{}z{}}}), so the search can walk the original target without changing
any offsets.
* The scan loop compares each target byte against both the lowercase and
uppercase forms of the pattern's first byte, so it does no case folding itself.
The comparison is almost always false, so the branch predicts well. Folding a
byte per position instead would put a data-dependent branch in the loop, which
mispredicts about half the time on mixed-case text.
* Only after a first-byte hit does the inner loop compare the remaining bytes
with both sides folded to lower case.
* Two small helpers, {{foldAsciiLowerCase}} and {{{}foldAsciiUpperCase{}}}, do
the folding.
The search has the same shape as {{{}UTF8String.contains{}}}: a first-byte
prefilter followed by a byte-wise match. The non-ASCII slow path is unchanged.
h3. Why are the changes needed?
Case-insensitive substring search over a UTF8_LCASE column, such as
{{{}contains(log_line, 'error'){}}}, runs once per row. On the ASCII path the
old code allocated a lowercased copy of every row's string even though case
folding needs no copy. It also lowercased the entire string before searching,
so a match near the start of a long string still paid to lowercase all of it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
