cloud-fan commented on code in PR #57450: URL: https://github.com/apache/spark/pull/57450#discussion_r3756701800
########## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala: ########## @@ -969,6 +969,62 @@ case class TryValidateUTF8(input: Expression) extends RuntimeReplaceable with Im } +/** + * A function that returns the Unicode normalization of a string. + */ +// scalastyle:off +@ExpressionDescription( + usage = """ + _FUNC_(str[, form]) - Returns the Unicode normalization of `str` using the normalization `form`. + Valid forms are 'NFC' (default), 'NFD', 'NFKC', and 'NFKD'. The form name is case-insensitive. Review Comment: **Blocking:** Please define normalization as a Spark semantic instead of inheriting whichever Unicode tables the current JDK ships. Spell out the canonical/compatibility decomposition and composition behavior for each form, and pin or otherwise document the Unicode-version policy with a version-sensitive conformance test. Otherwise the same query can return different text across supported JVM runtimes as their Unicode data evolves. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
