andygrove opened a new issue, #366:
URL: https://github.com/apache/datafusion-spark/issues/366

   ### Describe the bug
   
   `url_decode` raises an error when the decoded bytes are not valid UTF-8, and 
`try_url_decode` returns NULL. Spark decodes with Java's `URLDecoder`, which 
replaces each invalid byte with U+FFFD.
   
   ### To Reproduce
   
   ```sql
   SELECT url_decode('%FF'), try_url_decode('%FF');
   ```
   
   datafusion-spark raises `Invalid UTF-8 sequence` for `url_decode`, and 
`try_url_decode` returns `NULL`.
   
   In Spark 4.2.0, both return `U+FFFD` (`hex` gives `EFBFBD`).
   
   ### Expected behavior
   
   U+FFFD for each invalid byte from both functions, as in Spark. They still 
raise an error, or return NULL, for a malformed escape such as `%G1`.
   
   ### Additional context
   
   The records are in `url/url_decode.slt` and `url/try_url_decode.slt`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to