David Mollitor created SPARK-59557:
--------------------------------------
Summary: Avoid intermediate java.lang.String when casting integral
and boolean values to string
Key: SPARK-59557
URL: https://issues.apache.org/jira/browse/SPARK-59557
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Summary
Casting an integral (byte/short/int/long) or boolean value to string currently
routes through {{{}java.lang.String{}}}, allocating several throwaway objects
per row.
Both the interpreted and codegen cast paths in {{ToStringBase}} emit
{{UTF8String.fromString(String.valueOf(x))}} (codegen) /
\{{UTF8String.fromString(x.toString)}} (interpreted).
{{UTF8String.fromString(String)}} then does
{{{}str.getBytes(StandardCharsets.UTF_8){}}}, so a single {{CAST(<long> AS
STRING) }}allocates:
* a {{java.lang.String}} (from {{{}Long.toString{}}}),
* that String's backing {{{}byte[]{}}},
* a re-encoded UTF-8 {{byte[]}} (from {{{}getBytes{}}}), and
* the result {{{}UTF8String{}}}.
Only the last two are essential. {{CAST(<integral> AS STRING)}} is a very common
expression (key building, string concatenation, display), so the redundant
{{java.lang.String}} round-trip is broad, avoidable per-row overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
