[
https://issues.apache.org/jira/browse/SPARK-59952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59952:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid allocating lowercased copies in UTF8_LCASE contains for ASCII strings
> ---------------------------------------------------------------------------
>
> Key: SPARK-59952
> URL: https://issues.apache.org/jira/browse/SPARK-59952
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: David Mollitor
> Priority: Minor
> Labels: pull-request-available
>
> h3. What changes were proposed in this pull request?
> For the UTF8_LCASE collation, {{CollationSupport.Contains}} calls
> {{{}CollationAwareUTF8String.lowercaseContains{}}}. When both operands are
> full ASCII, its fast path ran
> {{{}target.toLowerCase().contains(pattern.toLowerCase()){}}}, which builds a
> lowercased copy of the whole target (a new byte array plus a new
> {{{}UTF8String{}}}) and of the pattern, searches the copies, and then throws
> them away.
> This PR replaces that fast path with {{{}lowercaseContainsAscii{}}}, which
> searches the original bytes and folds case as it compares, so it allocates
> nothing:
> * ASCII case folding maps each byte to one byte ({{{}A{}}}\-{{Z}} to
> {{a}}-{{{}z{}}}), so the search can walk the original target without changing
> any offsets.
> * The scan loop compares each target byte against both the lowercase and
> uppercase forms of the pattern's first byte, so it does no case folding
> itself. The comparison is almost always false, so the branch predicts well.
> Folding a byte per position instead would put a data-dependent branch in the
> loop, which mispredicts about half the time on mixed-case text.
> * Only after a first-byte hit does the inner loop compare the remaining
> bytes with both sides folded to lower case.
> * Two small helpers, {{foldAsciiLowerCase}} and {{{}foldAsciiUpperCase{}}},
> do the folding.
> The search has the same shape as {{{}UTF8String.contains{}}}: a first-byte
> prefilter followed by a byte-wise match. The non-ASCII slow path is unchanged.
> h3. Why are the changes needed?
> Case-insensitive substring search over a UTF8_LCASE column, such as
> {{{}contains(log_line, 'error'){}}}, runs once per row. On the ASCII path the
> old code allocated a lowercased copy of every row's string even though case
> folding needs no copy. It also lowercased the entire string before searching,
> so a match near the start of a long string still paid to lowercase all of it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
