SreeramaYeshwanthGowd opened a new pull request, #57466:
URL: https://github.com/apache/spark/pull/57466
### What changes were proposed in this pull request?
Add a matched pair of built in scalar functions:
- `to_base32(binary)` encodes bytes as an RFC 4648 Base32 string.
- `from_base32(string)` decodes an RFC 4648 Base32 string back to binary.
Spark already ships `hex`/`unhex` (base16) and `base64`/`unbase64`, so this
completes the standard base16/base32/base64 encoding ladder.
API surface added:
- SQL: `to_base32(binary)` and `from_base32(string)`
- Scala DataFrame: `functions.to_base32(col)` and
`functions.from_base32(col)`
- PySpark, classic and Spark Connect: `pyspark.sql.functions.to_base32` and
`from_base32`
Key design choices:
- Implemented as `RuntimeReplaceable` expressions backed by a
`StaticInvoke`, mirroring the existing `base64`/`unbase64` functions, so there
is no hand written codegen.
- The Base32 codec delegates to `org.apache.commons.codec.binary.Base32`
(RFC 4648), the same way `base64` delegates to `java.util.Base64`.
`java.util.Base64` has no Base32 support, and commons-codec is already a
dependency of the catalyst module, so this adds no new dependency.
- The SQL names `to_base32` and `from_base32` match Trino and BigQuery, and
follow Spark's established `to_*`/`from_*` conversion family (for example
`to_json`/`from_json`, `to_binary`). No major engine uses `base32`/`unbase32`.
- `from_base32` decodes leniently, ignoring characters outside the Base32
alphabet, consistent with the existing `unbase64` behavior. A strict variant is
out of scope, matching how `unbase64` defers strict decoding to `to_binary`.
- No SparkR binding is added, consistent with all recently added functions
(for example `to_binary`, `is_valid_utf8`, `luhn_check`), since SparkR is
deprecated as of Spark 4.0 and its function surface is frozen.
### Why are the changes needed?
Base32 (RFC 4648) is the standard encoding for TOTP and 2FA shared secrets
(RFC 6238), DNS NSEC3 records, and various identifier schemes. Spark has base16
(`hex`) and base64 but no base32, so users fall back to UDFs. Trino
(`to_base32`/`from_base32`) and Google BigQuery (`TO_BASE32`/`FROM_BASE32`)
both ship the pair, so this also improves parity with the engines Spark users
migrate from.
### Does this PR introduce _any_ user-facing change?
Yes. It adds two new built in SQL functions, `to_base32` and `from_base32`,
and the corresponding Scala and PySpark DataFrame API entries. No existing
behavior changes.
Example:
```
spark-sql> SELECT to_base32(encode('foobar', 'utf-8'));
MZXW6YTBOI======
spark-sql> SELECT cast(from_base32('MZXW6YTBOI======') as string);
foobar
```
### How was this patch tested?
Added catalyst unit tests in `StringExpressionsSuite` covering the RFC 4648
test vectors, round trip encode and decode, empty input, and null propagation,
and a DataFrame API test in `StringFunctionsSuite`. Added PySpark doctests for
both functions. Regenerated `sql-expression-schema.md`.
### Was this patch authored or co-authored using generative AI tooling?
<!-- Complete this section before submitting the PR. --> No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
