TechWizard9999 commented on code in PR #372:
URL: https://github.com/apache/datafusion-spark/pull/372#discussion_r4629152170


##########
crates/spark/src/function/url/url_decode.rs:
##########
@@ -79,23 +79,21 @@ impl UrlDecode {
     ///
     /// * `Ok(Cow<str>)` - The decoded string, borrowed from `value` when there
     ///   was nothing to rewrite and owned otherwise
-    /// * `Err(DataFusionError)` - If the input is malformed or contains 
invalid UTF-8
+    /// * `Err(DataFusionError)` - If the input is malformed
     fn decode(value: &str) -> Result<Cow<'_, str>> {
         // Check if the string has valid percent encoding
         Self::validate_percent_encoding(value)?;
 
-        match Self::replace_plus(value.as_bytes()) {
+        // Bytes that are not valid UTF-8 become U+FFFD, like Java's 
`URLDecoder`
+        Ok(match Self::replace_plus(value.as_bytes()) {
             // No '+' was rewritten, so the decode can borrow from `value` 
itself.
-            Cow::Borrowed(bytes) => percent_decode(bytes)
-                .decode_utf8()
-                .map_err(|e| exec_datafusion_err!("Invalid UTF-8 sequence: 
{e}")),
+            Cow::Borrowed(bytes) => percent_decode(bytes).decode_utf8_lossy(),

Review Comment:
   Folded surrogates into one. Thanks.



##########
crates/sqllogictest/test_files/url/url_decode.slt:
##########
@@ -53,7 +53,15 @@ Spark SQL!
 query error DataFusion error: Execution error: Invalid percent\-encoding: 
invalid hex sequence '%3%' at position 5
 SELECT url_decode('https%3%2F%2Fspark.apache.org'::string);
 
-# Bytes that are not valid UTF-8 are an error
-# xfail spark: Spark replaces bytes that are not valid UTF-8 with U+FFFD, 
datafusion-spark raises an error or returns NULL 
(https://github.com/apache/datafusion-spark/issues/366)
-query error DataFusion error: Execution error: Invalid UTF\-8 sequence
+# Bytes that are not valid UTF-8 become U+FFFD, like Spark
+query T
 SELECT url_decode('%FF'::string);
+----
+�
+
+query T
+SELECT url_decode(column1) FROM VALUES ('a%FFb%C3%28'), ('a%C3%A9'), (NULL);

Review Comment:
   Switched to hex and added cases.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to