[
https://issues.apache.org/jira/browse/SPARK-59394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59394:
-----------------------------------
Labels: pull-request-available (was: )
> UnBase64: decode the input bytes directly instead of converting to a String
> per row
> -----------------------------------------------------------------------------------
>
> Key: SPARK-59394
> URL: https://issues.apache.org/jira/browse/SPARK-59394
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: David Mollitor
> Priority: Minor
> Labels: pull-request-available
>
> h2. Summary
> The {{UnBase64}} expression (SQL {{unbase64}} and {{{}try_to_binary(...,
> 'base64'){}}}) decodes a base64 string to bytes. Both its interpreted path
> and its generated code call {{.toString()}} on the input {{UTF8String}}
> before decoding, which decodes the whole input from UTF-8 into a Java
> {{String}} on every row purely to hand it to the decoder.
> {{java.util.Base64.Decoder}} accepts a byte array directly, so passing the
> raw UTF-8 bytes ({{{}UTF8String.getBytes(){}}}) removes that per-row
> {{String}} allocation and UTF-8 decode.
> h2. Why this is safe (no behavior change)
> {{getMimeDecoder().decode(String)}} internally converts the string to bytes
> with ISO-8859-1 before decoding. For valid base64 – which is always ASCII –
> those bytes are identical to the raw UTF-8 bytes. For any non-ASCII or
> otherwise non-base64 content, the MIME decoder ignores every byte outside the
> base64 alphabet ({{{}A-Za-z0-9+/={}}}) under both paths, so the decoded
> output is the same. The change therefore produces identical results for all
> inputs.
> h2. Does this PR introduce _any_ user-facing change?
> No.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
