[
https://issues.apache.org/jira/browse/SPARK-59951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated SPARK-59951:
-----------------------------------
Description:
h3. What changes were proposed in this pull request?
{{UTF8String.toLowerCase()}} / {{toUpperCase()}} dispatch to the ASCII fast
paths {{toLowerCaseAscii()}} / {{toUpperCaseAscii()}} once {{isFullAscii()}}
holds. This PR rewrites those fast paths:
* Convert bytes with direct ASCII arithmetic ({{A}}\-{{Z}} <\-> {{a}}\-{{z}} by
+/- 32) instead of routing each byte through an autoboxed {{Function<Character,
Character>}} and {{Character.toLowerCase}}/{{toUpperCase}} (a full
Unicode-table lookup). Over ASCII code points the two are identical.
* Allocate the result array lazily, only when a byte actually changes case,
bulk-copying the unchanged prefix with {{Platform.copyMemory}}. A string
already in the target case returns {{this}} with no allocation.
* Tag the converted result as full ASCII ({{IsFullAscii.FULL_ASCII}}), since
ASCII case folding stays ASCII, so downstream operations skip re-scanning for
ASCII-ness.
* Remove the now-unused {{convertAscii}} helper and its
{{java.util.function.Function}} import, and add an {{assert isFullAscii()}}
precondition to both methods (callers already gate on {{isFullAscii()}}).
Also adds {{UTF8StringBenchmark}} exercising {{toLowerCase}}/{{toUpperCase}}
over all-lowercase, all-uppercase, and mixed-case ASCII inputs.
was:
h3. What changes were proposed in this pull request?
{{UTF8String.toLowerCase()}} / {{toUpperCase()}} dispatch to the ASCII fast
paths {{toLowerCaseAscii()}} / {{toUpperCaseAscii()}} once {{isFullAscii()}}
holds. This PR rewrites those fast paths:
* Convert bytes with direct ASCII arithmetic ({{A}}-{{Z}} <-> {{a}}-{{z}} by
+/- 32) instead of routing each byte through an autoboxed {{Function<Character,
Character>}} and {{Character.toLowerCase}}/{{toUpperCase}} (a full
Unicode-table lookup). Over ASCII code points the two are identical.
* Allocate the result array lazily, only when a byte actually changes case,
bulk-copying the unchanged prefix with {{Platform.copyMemory}}. A string
already in the target case returns {{this}} with no allocation.
* Tag the converted result as full ASCII ({{IsFullAscii.FULL_ASCII}}), since
ASCII case folding stays ASCII, so downstream operations skip re-scanning for
ASCII-ness.
* Remove the now-unused {{convertAscii}} helper and its
{{java.util.function.Function}} import, and add an {{assert isFullAscii()}}
precondition to both methods (callers already gate on {{isFullAscii()}}).
Also adds {{UTF8StringBenchmark}} exercising {{toLowerCase}}/{{toUpperCase}}
over all-lowercase, all-uppercase, and mixed-case ASCII inputs.
> Avoid allocation and boxing in UTF8String ASCII case conversion
> ---------------------------------------------------------------
>
> Key: SPARK-59951
> URL: https://issues.apache.org/jira/browse/SPARK-59951
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: David Mollitor
> Priority: Minor
>
> h3. What changes were proposed in this pull request?
> {{UTF8String.toLowerCase()}} / {{toUpperCase()}} dispatch to the ASCII fast
> paths {{toLowerCaseAscii()}} / {{toUpperCaseAscii()}} once {{isFullAscii()}}
> holds. This PR rewrites those fast paths:
> * Convert bytes with direct ASCII arithmetic ({{A}}\-{{Z}} <\-> {{a}}\-{{z}}
> by +/- 32) instead of routing each byte through an autoboxed
> {{Function<Character, Character>}} and
> {{Character.toLowerCase}}/{{toUpperCase}} (a full Unicode-table lookup). Over
> ASCII code points the two are identical.
> * Allocate the result array lazily, only when a byte actually changes case,
> bulk-copying the unchanged prefix with {{Platform.copyMemory}}. A string
> already in the target case returns {{this}} with no allocation.
> * Tag the converted result as full ASCII ({{IsFullAscii.FULL_ASCII}}), since
> ASCII case folding stays ASCII, so downstream operations skip re-scanning for
> ASCII-ness.
> * Remove the now-unused {{convertAscii}} helper and its
> {{java.util.function.Function}} import, and add an {{assert isFullAscii()}}
> precondition to both methods (callers already gate on {{isFullAscii()}}).
> Also adds {{UTF8StringBenchmark}} exercising {{toLowerCase}}/{{toUpperCase}}
> over all-lowercase, all-uppercase, and mixed-case ASCII inputs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
