Wali05 opened a new pull request, #26947:
URL: https://github.com/apache/datafusion/pull/26947
## Which issue does this PR close?
- Closes #26340.
## Rationale for this change
`unbase64` now decodes like Spark's `java.util.Base64.getMimeDecoder()`:
- Missing padding is accepted: `unbase64('123')` returns `D76D` instead of
an error, and `'YQ'` returns `61`.
- Bytes outside the base64 alphabet are skipped: `'YW Jj'` and `'YWJj!!'`
return `abc`, and `'!!!!'` returns an empty binary.
- `'YQ='` (two characters, one `=`) is now an error. main decoded it, but
Java rejects it.
- A dangling character (`'Y'`, `'YWJjZ'`) is still an error, with messages
modelled on Java's.
## What changes are included in this PR?
Before, `unbase64` was rewritten to `decode(x, 'base64pad')`. It now has its
own kernel with a small table-driven decoder that follows Java's MIME decode
loop. Binary, LargeBinary, BinaryView and FixedSizeBinary inputs keep their
current return types.
I did not use the `base64` crate here. It can't skip characters outside the
alphabet, and its padding modes don't match Java (it accepts `'YQ='` and
rejects `'YQ==='`, which is the opposite of Java).
## What is the testing strategy for this PR?
- `spark/string/base64.slt`: the two cases that expected `unbase64('123')`
to fail now expect the Spark results from the issue. New cases cover skipped
characters, empty input, a dangling character, bad padding and FixedSizeBinary
input. On main this file fails with 7 query errors. With the change it passes.
- Unit test for the decoder in `base64.rs`.
- On 1M padded strings, unbase64 takes 43.9 ms on main and 46.8 ms here
(median, release build).
The valid outputs come from the Spark 4.0.0 results in the issue. The error
cases follow the checks in Java's decoder.
## Are there any user-facing changes?
Yes. `unbase64` now returns results for input that Spark decodes and used to
error, and it rejects `'YQ='`, which used to decode. Spark goes through the
same Java MIME decoder, which rejects `'YQ='` (checked on JDK 21, not run on
Spark). No API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
