David Mollitor created SPARK-59951:
--------------------------------------
Summary: Avoid allocation and boxing in UTF8String ASCII case
conversion
Key: SPARK-59951
URL: https://issues.apache.org/jira/browse/SPARK-59951
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: David Mollitor
h3. What changes were proposed in this pull request?
{{UTF8String.toLowerCase()}} / {{toUpperCase()}} dispatch to the ASCII fast
paths {{toLowerCaseAscii()}} / {{toUpperCaseAscii()}} once {{isFullAscii()}}
holds. This PR rewrites those fast paths:
* Convert bytes with direct ASCII arithmetic ({{A}}-{{Z}} <-> {{a}}-{{z}} by
+/- 32) instead of routing each byte through an autoboxed {{Function<Character,
Character>}} and {{Character.toLowerCase}}/{{toUpperCase}} (a full
Unicode-table lookup). Over ASCII code points the two are identical.
* Allocate the result array lazily, only when a byte actually changes case,
bulk-copying the unchanged prefix with {{Platform.copyMemory}}. A string
already in the target case returns {{this}} with no allocation.
* Tag the converted result as full ASCII ({{IsFullAscii.FULL_ASCII}}), since
ASCII case folding stays ASCII, so downstream operations skip re-scanning for
ASCII-ness.
* Remove the now-unused {{convertAscii}} helper and its
{{java.util.function.Function}} import, and add an {{assert isFullAscii()}}
precondition to both methods (callers already gate on {{isFullAscii()}}).
Also adds {{UTF8StringBenchmark}} exercising {{toLowerCase}}/{{toUpperCase}}
over all-lowercase, all-uppercase, and mixed-case ASCII inputs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
