andygrove opened a new issue, #366:
URL: https://github.com/apache/datafusion-spark/issues/366
### Describe the bug
`url_decode` raises an error when the decoded bytes are not valid UTF-8, and
`try_url_decode` returns NULL. Spark decodes with Java's `URLDecoder`, which
replaces each invalid byte with U+FFFD.
### To Reproduce
```sql
SELECT url_decode('%FF'), try_url_decode('%FF');
```
datafusion-spark raises `Invalid UTF-8 sequence` for `url_decode`, and
`try_url_decode` returns `NULL`.
In Spark 4.2.0, both return `U+FFFD` (`hex` gives `EFBFBD`).
### Expected behavior
U+FFFD for each invalid byte from both functions, as in Spark. They still
raise an error, or return NULL, for a malformed escape such as `%G1`.
### Additional context
The records are in `url/url_decode.slt` and `url/try_url_decode.slt`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]