manuzhang commented on code in PR #4763: URL: https://github.com/apache/datafusion-comet/pull/4763#discussion_r3578466554
########## docs/source/contributor-guide/adding_a_new_expression.md: ########## @@ -478,6 +478,21 @@ pub fn spark_ceil( } ``` +#### Producing strings from arbitrary bytes + +Spark's `StringType` may contain bytes that are not valid UTF-8 (Spark stores them verbatim), but +Arrow's string arrays must be valid UTF-8. Any native code that turns arbitrary bytes into a string +must therefore **decode** them, not reinterpret them. Building a Rust `String`/`&str` from non-UTF-8 +bytes is undefined behaviour, and an Arrow string array that holds invalid UTF-8 is unsound for the +downstream kernels that read it via `StringArray::value` (which assumes the UTF-8 invariant). + +Use `crate::utils::decode_utf8_spark_lossy`. It replaces each ill-formed sequence with `U+FFFD` Review Comment: `decode_utf8_spark_lossy` was moved to `datafusion-comet-common` and is not exported from this crate’s `utils` module. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
