ozenyunus35 opened a new pull request, #26345:
URL: https://github.com/apache/datafusion/pull/26345
## Which issue does this PR close?
- Closes #26340.
## Rationale for this change
Spark's `unbase64` decodes with Java's MIME Base64 decoder, which accepts
missing padding and skips characters outside the Base64 alphabet.
`datafusion-spark` rewrote `unbase64` to the core `decode(..., 'base64pad')`,
so inputs like `'123'`, `'YQ'` or `'YW Jj'` failed here while Spark decodes
them.
## What changes are included in this PR?
`unbase64` no longer simplifies to `decode`; it decodes each value with a
small `mime_decode` function:
- characters outside the standard alphabet (`A-Z a-z 0-9 + /`) are skipped,
and padding is optional;
- a final unit of one character is an error ("last unit does not have enough
valid bits");
- `=` must complete the current unit (`xx==` or `xxx=`), and only
non-alphabet characters may follow it; anything else is an "invalid padding"
error.
Binary, LargeBinary and BinaryView inputs are handled; the return type is
unchanged (LargeBinary for LargeBinary, Binary otherwise). `base64` is
unchanged.
I took the expected results from running
`java.util.Base64.getMimeDecoder().decode(...)` on JDK 21 for each input in the
tests, including the edge cases around padding (`YQ=`, `YQ= =`, `YQ===`,
`YQ==x`, `YWJj=`, `YQ==YQ==`).
## What is the testing strategy for this PR?
- `base64.slt`: `unbase64('123')` (string and binary) now returns `d76d`
instead of an error, as the issue asks, plus cases for missing padding, skipped
characters, an input with only skipped characters, `xxx=`, a LargeBinary input,
and the three error cases. Six of these fail on `main`.
- Unit tests for `mime_decode` cover the accepted inputs and the error cases
above.
- `cargo test -p datafusion-spark --lib string::base64`, the
`spark/string/base64` sqllogictests and `cargo clippy -p datafusion-spark
--all-targets -- -D warnings` pass.
## Are there any user-facing changes?
Yes: `unbase64` now accepts the inputs Spark accepts (missing padding,
characters outside the alphabet), instead of failing.
I used an AI assistant to help draft parts of the code and tests; I reviewed
the changes and ran the tests myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
