This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2572d806923a [SPARK-58275][SQL][PYTHON] Add `normalize` Unicode
normalization SQL function
2572d806923a is described below
commit 2572d806923a9cb5f2094a822d4fba9df979685b
Author: SreeramaYeshwanthGowd <[email protected]>
AuthorDate: Tue Aug 11 22:40:24 2026 +0800
[SPARK-58275][SQL][PYTHON] Add `normalize` Unicode normalization SQL
function
### What changes were proposed in this pull request?
Add a built in `normalize(str[, form])` scalar function that performs
Unicode normalization, backed by Spark's bundled ICU4J library.
API surface added:
- SQL: `normalize(str)` and `normalize(str, form)`
- Scala DataFrame: `functions.normalize(col)` and `functions.normalize(col,
form)`
- PySpark, classic and Spark Connect: `pyspark.sql.functions.normalize(str,
form=None)`
Key design choices:
- The default form is NFC. Supported forms are NFC, NFD, NFKC, and NFKD,
case insensitive, matching the ANSI SQL standard and PostgreSQL, Trino, and
BigQuery. Forms follow Unicode Standard Annex #15's decomposition/composition
algorithm.
- The `form` argument is a regular string expression or column, not
required to be foldable, consistent with how other Spark functions take a mode
string (for example `date_trunc`). This keeps the change small and idiomatic
and avoids adding new grammar. An invalid form raises a clear
`INVALID_PARAMETER_VALUE.NORMALIZE_FORM` error.
- Implemented as a `RuntimeReplaceable` expression backed by a
`StaticInvoke` into `ExpressionImplUtils`, so there is no hand written codegen,
following the recent `hmac` function.
- Normalization uses ICU4J (the same library backing Spark's collation
support) rather than the JDK's `java.text.Normalizer`, so behavior is pinned to
Spark's bundled ICU4J/Unicode data and does not vary across JVM vendors or
versions. This is documented on the public SQL/Scala/PySpark doc surfaces.
### Why are the changes needed?
Unicode normalization is a common need in text processing and data
cleaning, for example to compare strings that look identical but differ in code
point composition. Spark has no built in way to do this, so users fall back to
a UDF, which cannot be optimized by Catalyst. This function is part of the SQL
standard and is already available in PostgreSQL, Trino, and Google BigQuery, so
it also improves parity with the engines that Spark users migrate from.
### Does this PR introduce _any_ user-facing change?
Yes. It adds a new built in SQL function `normalize` and the corresponding
Scala and PySpark DataFrame API entries. No existing behavior changes.
Example:
```
spark-sql> SELECT normalize('fi', 'NFKC');
fi
```
### How was this patch tested?
Added unit tests in `ExpressionImplUtilsSuite` and `StringExpressionsSuite`
covering all four forms (NFC, NFD, NFKC, NFKD), a case insensitive form name,
null propagation, empty input, a supplementary (surrogate pair) character, a
canonical singleton decomposition, a compatibility decomposition, the invalid
form error, and a conformance vector for canonical reordering of combining
marks with different combining classes (Unicode Standard Annex #15). Added a
DataFrame API test in `Strin [...]
### Was this patch authored or co-authored using generative AI tooling? No
Closes #57450 from SreeramaYeshwanthGowd/add-normalize-function.
Authored-by: SreeramaYeshwanthGowd <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 5 ++
.../source/reference/pyspark.sql/functions.rst | 1 +
python/pyspark/sql/connect/functions/builtin.py | 10 ++++
python/pyspark/sql/functions/__init__.py | 1 +
python/pyspark/sql/functions/builtin.py | 40 +++++++++++++++
python/pyspark/sql/tests/test_functions.py | 8 +++
.../scala/org/apache/spark/sql/functions.scala | 28 ++++++++++
.../catalyst/expressions/ExpressionImplUtils.java | 24 +++++++++
.../sql/catalyst/analysis/FunctionRegistry.scala | 1 +
.../catalyst/expressions/stringExpressions.scala | 60 ++++++++++++++++++++++
.../spark/sql/errors/QueryExecutionErrors.scala | 9 ++++
.../expressions/ExpressionImplUtilsSuite.scala | 42 +++++++++++++++
.../expressions/StringExpressionsSuite.scala | 23 ++++++++-
.../sql-functions/sql-expression-schema.md | 1 +
.../nonansi/string-functions.sql.out | 35 +++++++++++++
.../analyzer-results/string-functions.sql.out | 35 +++++++++++++
.../sql-tests/inputs/string-functions.sql | 9 +++-
.../results/nonansi/string-functions.sql.out | 49 ++++++++++++++++++
.../sql-tests/results/string-functions.sql.out | 49 ++++++++++++++++++
.../apache/spark/sql/StringFunctionsSuite.scala | 10 ++++
20 files changed, 438 insertions(+), 2 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index 62eef58c0543..f89f3bf5e445 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -4782,6 +4782,11 @@
"expects a long literal, but got <invalidValue>."
]
},
+ "NORMALIZE_FORM" : {
+ "message" : [
+ "expects one of the normalization forms 'NFC', 'NFD', 'NFKC',
'NFKD', but got <form>."
+ ]
+ },
"NULL" : {
"message" : [
"expects a non-NULL value."
diff --git a/python/docs/source/reference/pyspark.sql/functions.rst
b/python/docs/source/reference/pyspark.sql/functions.rst
index dd9231b8dd4e..a79922c8051d 100644
--- a/python/docs/source/reference/pyspark.sql/functions.rst
+++ b/python/docs/source/reference/pyspark.sql/functions.rst
@@ -193,6 +193,7 @@ String Functions
ltrim
make_valid_utf8
mask
+ normalize
octet_length
overlay
position
diff --git a/python/pyspark/sql/connect/functions/builtin.py
b/python/pyspark/sql/connect/functions/builtin.py
index ea2e2fafc501..8451ae386770 100644
--- a/python/pyspark/sql/connect/functions/builtin.py
+++ b/python/pyspark/sql/connect/functions/builtin.py
@@ -2741,6 +2741,16 @@ def try_validate_utf8(str: "ColumnOrName") -> Column:
try_validate_utf8.__doc__ = pysparkfuncs.try_validate_utf8.__doc__
+def normalize(str: "ColumnOrName", form: Optional["ColumnOrName"] = None) ->
Column:
+ if form is None:
+ return _invoke_function_over_columns("normalize", str)
+ else:
+ return _invoke_function_over_columns("normalize", str, form)
+
+
+normalize.__doc__ = pysparkfuncs.normalize.__doc__
+
+
def format_number(col: "ColumnOrName", d: int) -> Column:
return _invoke_function("format_number", _to_col(col), lit(d))
diff --git a/python/pyspark/sql/functions/__init__.py
b/python/pyspark/sql/functions/__init__.py
index a66d7d36d103..d20790f20bcb 100644
--- a/python/pyspark/sql/functions/__init__.py
+++ b/python/pyspark/sql/functions/__init__.py
@@ -154,6 +154,7 @@ __all__ = [ # noqa: F405
"ltrim",
"make_valid_utf8",
"mask",
+ "normalize",
"octet_length",
"overlay",
"position",
diff --git a/python/pyspark/sql/functions/builtin.py
b/python/pyspark/sql/functions/builtin.py
index dff38c122ca0..d5ec94fa6d62 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -16024,6 +16024,46 @@ def try_validate_utf8(str: "ColumnOrName") -> Column:
return _invoke_function_over_columns("try_validate_utf8", str)
+@_try_remote_functions
+def normalize(str: "ColumnOrName", form: Optional["ColumnOrName"] = None) ->
Column:
+ """
+ Returns the Unicode normalization of ``str`` using the given normalization
``form``, as
+ defined by Unicode Standard Annex #15. Normalization is backed by Spark's
bundled ICU4J
+ library rather than the JVM's own Unicode data, so results are stable
across JVM vendors
+ and versions.
+
+ .. versionadded:: 4.4.0
+
+ Parameters
+ ----------
+ str : :class:`~pyspark.sql.Column` or column name
+ the input string to normalize.
+ form : :class:`~pyspark.sql.Column` or column name, optional
+ the normalization form, one of 'NFC', 'NFD', 'NFKC', 'NFKD'
(case-insensitive).
+ If omitted, 'NFC' is used.
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ the normalized string.
+
+ Examples
+ --------
+ >>> import pyspark.sql.functions as sf
+ >>> df = spark.createDataFrame([("\ufb01",)], ["s"])
+ >>> df.select(sf.normalize(df.s, sf.lit("NFKC"))).show()
+ +------------------+
+ |normalize(s, NFKC)|
+ +------------------+
+ | fi|
+ +------------------+
+ """
+ if form is None:
+ return _invoke_function_over_columns("normalize", str)
+ else:
+ return _invoke_function_over_columns("normalize", str, form)
+
+
@_try_remote_functions
def format_number(col: "ColumnOrName", d: int) -> Column:
"""
diff --git a/python/pyspark/sql/tests/test_functions.py
b/python/pyspark/sql/tests/test_functions.py
index b5ee99fc9565..61b4c4d258a0 100644
--- a/python/pyspark/sql/tests/test_functions.py
+++ b/python/pyspark/sql/tests/test_functions.py
@@ -1022,6 +1022,14 @@ class FunctionsTestsMixin:
null_result = df.select(F.jaro_winkler_similarity(df.l,
F.lit(None))).first()[0]
self.assertIsNone(null_result)
+ def test_normalize_function(self):
+ df = self.spark.createDataFrame([("\ufb01",)], ["s"])
+ result = df.select(F.normalize(df.s, F.lit("NFKC"))).first()[0]
+ self.assertEqual(result, "fi")
+ # Null handling
+ null_result = df.select(F.normalize(F.lit(None),
F.lit("NFC"))).first()[0]
+ self.assertIsNone(null_result)
+
def test_between_function(self):
df = self.spark.createDataFrame(
[Row(a=1, b=2, c=3), Row(a=2, b=1, c=3), Row(a=4, b=1, c=4)]
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
b/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
index 514ca368e6a3..96d5ec930604 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
@@ -7533,6 +7533,34 @@ object functions {
def try_validate_utf8(str: Column): Column =
Column.fn("try_validate_utf8", str)
+ /**
+ * Returns the Unicode normalization of `str` using the given normalization
`form`. Valid forms
+ * are 'NFC', 'NFD', 'NFKC', and 'NFKD', as defined by Unicode Standard
Annex #15. The form name
+ * is case-insensitive. Normalization is backed by Spark's bundled ICU4J
library rather than the
+ * JVM's own Unicode data, so results are stable across JVM vendors and
versions.
+ *
+ * @param str
+ * the input string to normalize.
+ * @param form
+ * the normalization form: 'NFC', 'NFD', 'NFKC', or 'NFKD'.
+ * @group string_funcs
+ * @since 4.4.0
+ */
+ def normalize(str: Column, form: Column): Column =
+ Column.fn("normalize", str, form)
+
+ /**
+ * Returns the Unicode normalization of `str` using the default form 'NFC'.
To use a different
+ * form, call the two-argument overload.
+ *
+ * @param str
+ * the input string to normalize.
+ * @group string_funcs
+ * @since 4.4.0
+ */
+ def normalize(str: Column): Column =
+ Column.fn("normalize", str)
+
/**
* Formats numeric column x to a format like '#,###,###.##', rounded to d
decimal places with
* HALF_EVEN round mode, and returns the result as a string column.
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java
index 3f2c7e1fc5d4..4f7446d0cd9c 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java
@@ -17,6 +17,8 @@
package org.apache.spark.sql.catalyst.expressions;
+import com.ibm.icu.text.Normalizer2;
+
import java.nio.ByteBuffer;
import java.security.GeneralSecurityException;
import java.security.SecureRandom;
@@ -149,6 +151,28 @@ public class ExpressionImplUtils {
else return null;
}
+ /**
+ * Normalizes the given string using the given Unicode normalization form,
per the
+ * decomposition/composition algorithm in Unicode Standard Annex #15. Uses
ICU4J, the same
+ * library backing Spark's collation support, instead of the JDK's {@code
java.text.Normalizer}
+ * so the result is pinned to Spark's bundled ICU4J/Unicode data (see {@code
icu4j.version} in
+ * pom.xml) and does not vary across JVM vendors or versions.
+ *
+ * @param input the input string to normalize.
+ * @param form the normalization form, one of NFC, NFD, NFKC, NFKD
(case-insensitive).
+ * @return the normalized string.
+ */
+ public static UTF8String normalize(UTF8String input, UTF8String form) {
+ Normalizer2 normalizer = switch (form.toString().toUpperCase(Locale.ROOT))
{
+ case "NFC" -> Normalizer2.getNFCInstance();
+ case "NFD" -> Normalizer2.getNFDInstance();
+ case "NFKC" -> Normalizer2.getNFKCInstance();
+ case "NFKD" -> Normalizer2.getNFKDInstance();
+ default -> throw
QueryExecutionErrors.invalidNormalizeFormError(form.toString());
+ };
+ return UTF8String.fromString(normalizer.normalize(input.toString()));
+ }
+
public static byte[] aesEncrypt(byte[] input,
byte[] key,
UTF8String mode,
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
index d7b0f4a440d1..5eef532ab20b 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
@@ -691,6 +691,7 @@ object FunctionRegistry {
expression[ValidateUTF8]("validate_utf8"),
expression[TryValidateUTF8]("try_validate_utf8"),
expression[Quote]("quote"),
+ expression[Normalize]("normalize"),
// url functions
expression[UrlEncode]("url_encode"),
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
index 6dfdea1bdc7b..45c9e9256aed 100755
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
@@ -969,6 +969,66 @@ case class TryValidateUTF8(input: Expression) extends
RuntimeReplaceable with Im
}
+/**
+ * A function that returns the Unicode normalization of a string.
+ */
+// scalastyle:off
+@ExpressionDescription(
+ usage = """
+ _FUNC_(str[, form]) - Returns the Unicode normalization of `str` using the
normalization `form`.
+ Valid forms are 'NFC' (default), 'NFD', 'NFKC', and 'NFKD', as defined
by Unicode Standard
+ Annex #15: 'NFD'/'NFKD' apply canonical/compatibility decomposition;
'NFC'/'NFKC' apply the
+ same decomposition followed by canonical composition. The form name is
case-insensitive.
+ Normalization is backed by Spark's bundled ICU4J library rather than the
JVM's own Unicode
+ data, so results are stable across JVM vendors and versions.
+ """,
+ arguments = """
+ Arguments:
+ * str - a string expression to normalize.
+ * form - a string expression giving the normalization form: 'NFC',
'NFD', 'NFKC', or 'NFKD'.
+ If omitted, 'NFC' is used.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_('fi', 'NFKC');
+ fi
+ """,
+ since = "4.4.0",
+ group = "string_funcs")
+// scalastyle:on
+case class Normalize(input: Expression, form: Expression)
+ extends RuntimeReplaceable with ImplicitCastInputTypes with
BinaryLike[Expression] {
+ override def nullIntolerant: Boolean = true
+
+ override lazy val replacement: Expression =
+ StaticInvoke(
+ classOf[ExpressionImplUtils],
+ input.dataType,
+ "normalize",
+ Seq(input, form),
+ inputTypes)
+
+ def this(input: Expression) = this(input, Literal("NFC"))
+
+ override def inputTypes: Seq[AbstractDataType] =
+ Seq(StringTypeWithCollation(supportsTrimCollation = true),
+ StringTypeWithCollation(supportsTrimCollation = true))
+
+ override def nodeName: String = "normalize"
+
+ override def nullable: Boolean = true
+
+ override def left: Expression = input
+
+ override def right: Expression = form
+
+ override protected def withNewChildrenInternal(
+ newLeft: Expression, newRight: Expression): Normalize = {
+ copy(input = newLeft, form = newRight)
+ }
+
+}
+
/**
* Replace all occurrences with string.
*/
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
index 8f1b16232fd4..646d296c11c6 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
@@ -2566,6 +2566,15 @@ private[sql] object QueryExecutionErrors extends
QueryErrorsBase with ExecutionE
"detailMessage" -> detailMessage))
}
+ def invalidNormalizeFormError(form: String): RuntimeException = {
+ new SparkRuntimeException(
+ errorClass = "INVALID_PARAMETER_VALUE.NORMALIZE_FORM",
+ messageParameters = Map(
+ "parameter" -> toSQLId("form"),
+ "functionName" -> toSQLId("normalize"),
+ "form" -> toSQLValue(form, StringType)))
+ }
+
def hiveTableWithAnsiIntervalsError(
table: TableIdentifier): SparkUnsupportedOperationException = {
new SparkUnsupportedOperationException(
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtilsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtilsSuite.scala
index e3a9177bfdbc..608a13d47c59 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtilsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtilsSuite.scala
@@ -480,4 +480,46 @@ class ExpressionImplUtilsSuite extends SparkFunSuite {
tryValidateUTF8(UTF8String.fromBytes(Array[Byte](0xFF.toByte)), null)
}
+ // These vectors follow Unicode Standard Annex #15's
decomposition/composition algorithm and
+ // pin normalize() to Spark's bundled ICU4J/Unicode data (see icu4j.version
in pom.xml): a
+ // future ICU4J upgrade that changed these mappings would fail this test,
rather than silently
+ // changing query results.
+ test("Normalize with supported forms") {
+ def normalize(input: String, form: String): String =
+ ExpressionImplUtils.normalize(
+ UTF8String.fromString(input), UTF8String.fromString(form)).toString
+
+ // scalastyle:off nonascii
+ assert(normalize("\uFB01", "NFKC") == "fi")
+ assert(normalize("A\u030A", "NFC") == "\u00C5")
+ assert(normalize("\u00C5", "NFD") == "A\u030A")
+ assert(normalize("\uFB01", "nfkc") == "fi")
+ assert(normalize("\u00BD", "NFKD") == "1" + "\u2044" + "2")
+ assert(normalize("\u00BD", "NFD") == "\u00BD")
+ assert(normalize("\uD835\uDC00", "NFKC") == "A")
+ assert(normalize("", "NFC") == "")
+
+ // Combining marks with different combining classes are canonically
reordered before
+ // composition (UAX #15 canonical ordering algorithm), so applying
COMBINING DOT BELOW
+ // (U+0323) and COMBINING ACUTE ACCENT (U+0301) to the same base letter in
either input
+ // order must normalize (NFC) to the same result.
+ val dotBelow = "\u0323"
+ val acute = "\u0301"
+ assert(normalize("a" + dotBelow + acute, "NFC") == normalize("a" + acute +
dotBelow, "NFC"))
+ // scalastyle:on nonascii
+ }
+
+ test("Normalize invalid form error") {
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ ExpressionImplUtils.normalize(
+ UTF8String.fromString("abc"), UTF8String.fromString("NFE"))
+ },
+ condition = "INVALID_PARAMETER_VALUE.NORMALIZE_FORM",
+ parameters = Map(
+ "parameter" -> "`form`",
+ "functionName" -> "`normalize`",
+ "form" -> "'NFE'"))
+ }
+
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
index 4ae723f33b39..0cb944192cb8 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.catalyst.expressions
import java.math.{BigDecimal => JavaBigDecimal}
-import org.apache.spark.{SPARK_DOC_ROOT, SparkFunSuite,
SparkIllegalArgumentException}
+import org.apache.spark.{SPARK_DOC_ROOT, SparkFunSuite,
SparkIllegalArgumentException, SparkRuntimeException}
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import
org.apache.spark.sql.catalyst.analysis.TypeCheckResult.{DataTypeMismatch,
InvalidFormat}
@@ -2338,4 +2338,25 @@ class StringExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
assert(formatNumberCopy ne formatNumber,
"freshCopyIfContainsStatefulExpression should return a new instance for
FormatNumber")
}
+
+ test("Normalize") {
+ // scalastyle:off nonascii
+ checkEvaluation(new Normalize(Literal("A\u030A")), "\u00C5")
+ checkEvaluation(Normalize(Literal("A\u030A"), Literal("NFC")), "\u00C5")
+ checkEvaluation(Normalize(Literal("\u00C5"), Literal("NFD")), "A\u030A")
+ checkEvaluation(Normalize(Literal("\uFB01"), Literal("NFKC")), "fi")
+ // scalastyle:on nonascii
+ checkEvaluation(Normalize(Literal.create(null, StringType),
Literal("NFC")), null)
+ checkEvaluation(Normalize(Literal("abc"), Literal.create(null,
StringType)), null)
+ }
+
+ test("Normalize invalid form") {
+ checkErrorInExpression[SparkRuntimeException](
+ Normalize(Literal("abc"), Literal("NFE")),
+ "INVALID_PARAMETER_VALUE.NORMALIZE_FORM",
+ Map(
+ "parameter" -> "`form`",
+ "functionName" -> "`normalize`",
+ "form" -> "'NFE'"))
+ }
}
diff --git a/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
b/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
index 402070bc0d08..7c389ed360ab 100644
--- a/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
+++ b/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
@@ -261,6 +261,7 @@
| org.apache.spark.sql.catalyst.expressions.NaNvl | nanvl | SELECT
nanvl(cast('NaN' as double), 123) | struct<nanvl(CAST(NaN AS DOUBLE),
123):double> |
| org.apache.spark.sql.catalyst.expressions.NanosToTimestamp | timestamp_nanos
| SELECT timestamp_nanos(1230219000123456789) |
struct<timestamp_nanos(1230219000123456789):timestamp_ltz(9)> |
| org.apache.spark.sql.catalyst.expressions.NextDay | next_day | SELECT
next_day('2015-01-14', 'TU') | struct<next_day(2015-01-14, TU):date> |
+| org.apache.spark.sql.catalyst.expressions.Normalize | normalize | SELECT
normalize('fi', 'NFKC') | struct<normalize(fi, NFKC):string> |
| org.apache.spark.sql.catalyst.expressions.Not | ! | SELECT ! true |
struct<(NOT true):boolean> |
| org.apache.spark.sql.catalyst.expressions.Not | not | SELECT not true |
struct<(NOT true):boolean> |
| org.apache.spark.sql.catalyst.expressions.NowExpressionBuilder | now |
SELECT now() | struct<now():timestamp> |
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/nonansi/string-functions.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/nonansi/string-functions.sql.out
index 15e6f3ada266..64b21c7114a0 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/nonansi/string-functions.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/nonansi/string-functions.sql.out
@@ -2167,3 +2167,38 @@ select instr(null, null, cast(null as int), cast(null as
int))
-- !query analysis
Project [instr(cast(null as string), cast(null as string), cast(null as int),
cast(null as int)) AS instr(NULL, NULL, CAST(NULL AS INT), CAST(NULL AS INT))#x]
+- OneRowRelation
+
+
+-- !query
+select normalize('hello')
+-- !query analysis
+Project [normalize(hello, NFC) AS normalize(hello, NFC)#x]
++- OneRowRelation
+
+
+-- !query
+select normalize('hello', 'NFD')
+-- !query analysis
+Project [normalize(hello, NFD) AS normalize(hello, NFD)#x]
++- OneRowRelation
+
+
+-- !query
+select normalize('fi', 'NFKC')
+-- !query analysis
+Project [normalize(fi, NFKC) AS normalize(fi, NFKC)#x]
++- OneRowRelation
+
+
+-- !query
+select normalize(null, 'NFC')
+-- !query analysis
+Project [normalize(cast(null as string), NFC) AS normalize(NULL, NFC)#x]
++- OneRowRelation
+
+
+-- !query
+select normalize('hello', 'not_a_form')
+-- !query analysis
+Project [normalize(hello, not_a_form) AS normalize(hello, not_a_form)#x]
++- OneRowRelation
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/string-functions.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/string-functions.sql.out
index 15e6f3ada266..64b21c7114a0 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/string-functions.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/string-functions.sql.out
@@ -2167,3 +2167,38 @@ select instr(null, null, cast(null as int), cast(null as
int))
-- !query analysis
Project [instr(cast(null as string), cast(null as string), cast(null as int),
cast(null as int)) AS instr(NULL, NULL, CAST(NULL AS INT), CAST(NULL AS INT))#x]
+- OneRowRelation
+
+
+-- !query
+select normalize('hello')
+-- !query analysis
+Project [normalize(hello, NFC) AS normalize(hello, NFC)#x]
++- OneRowRelation
+
+
+-- !query
+select normalize('hello', 'NFD')
+-- !query analysis
+Project [normalize(hello, NFD) AS normalize(hello, NFD)#x]
++- OneRowRelation
+
+
+-- !query
+select normalize('fi', 'NFKC')
+-- !query analysis
+Project [normalize(fi, NFKC) AS normalize(fi, NFKC)#x]
++- OneRowRelation
+
+
+-- !query
+select normalize(null, 'NFC')
+-- !query analysis
+Project [normalize(cast(null as string), NFC) AS normalize(NULL, NFC)#x]
++- OneRowRelation
+
+
+-- !query
+select normalize('hello', 'not_a_form')
+-- !query analysis
+Project [normalize(hello, not_a_form) AS normalize(hello, not_a_form)#x]
++- OneRowRelation
diff --git a/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
b/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
index c432093e1ae8..4e4cb6a401f5 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
@@ -370,4 +370,11 @@ select instr('a', 'b', 1, cast(null as int));
select instr(null, 'b', 1);
select instr('a', null, 1);
select instr('a', 'b', cast(null as int), 2);
-select instr(null, null, cast(null as int), cast(null as int));
\ No newline at end of file
+select instr(null, null, cast(null as int), cast(null as int));
+
+-- normalize
+select normalize('hello');
+select normalize('hello', 'NFD');
+select normalize('fi', 'NFKC');
+select normalize(null, 'NFC');
+select normalize('hello', 'not_a_form');
\ No newline at end of file
diff --git
a/sql/core/src/test/resources/sql-tests/results/nonansi/string-functions.sql.out
b/sql/core/src/test/resources/sql-tests/results/nonansi/string-functions.sql.out
index 1a8521a37cfc..c6655b01d0e6 100644
---
a/sql/core/src/test/resources/sql-tests/results/nonansi/string-functions.sql.out
+++
b/sql/core/src/test/resources/sql-tests/results/nonansi/string-functions.sql.out
@@ -2701,3 +2701,52 @@ select instr(null, null, cast(null as int), cast(null as
int))
struct<instr(NULL, NULL, CAST(NULL AS INT), CAST(NULL AS INT)):int>
-- !query output
NULL
+
+
+-- !query
+select normalize('hello')
+-- !query schema
+struct<normalize(hello, NFC):string>
+-- !query output
+hello
+
+
+-- !query
+select normalize('hello', 'NFD')
+-- !query schema
+struct<normalize(hello, NFD):string>
+-- !query output
+hello
+
+
+-- !query
+select normalize('fi', 'NFKC')
+-- !query schema
+struct<normalize(fi, NFKC):string>
+-- !query output
+fi
+
+
+-- !query
+select normalize(null, 'NFC')
+-- !query schema
+struct<normalize(NULL, NFC):string>
+-- !query output
+NULL
+
+
+-- !query
+select normalize('hello', 'not_a_form')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkRuntimeException
+{
+ "errorClass" : "INVALID_PARAMETER_VALUE.NORMALIZE_FORM",
+ "sqlState" : "22023",
+ "messageParameters" : {
+ "form" : "'not_a_form'",
+ "functionName" : "`normalize`",
+ "parameter" : "`form`"
+ }
+}
diff --git
a/sql/core/src/test/resources/sql-tests/results/string-functions.sql.out
b/sql/core/src/test/resources/sql-tests/results/string-functions.sql.out
index b0497277c59b..e6be16af6784 100644
--- a/sql/core/src/test/resources/sql-tests/results/string-functions.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/string-functions.sql.out
@@ -2769,3 +2769,52 @@ select instr(null, null, cast(null as int), cast(null as
int))
struct<instr(NULL, NULL, CAST(NULL AS INT), CAST(NULL AS INT)):int>
-- !query output
NULL
+
+
+-- !query
+select normalize('hello')
+-- !query schema
+struct<normalize(hello, NFC):string>
+-- !query output
+hello
+
+
+-- !query
+select normalize('hello', 'NFD')
+-- !query schema
+struct<normalize(hello, NFD):string>
+-- !query output
+hello
+
+
+-- !query
+select normalize('fi', 'NFKC')
+-- !query schema
+struct<normalize(fi, NFKC):string>
+-- !query output
+fi
+
+
+-- !query
+select normalize(null, 'NFC')
+-- !query schema
+struct<normalize(NULL, NFC):string>
+-- !query output
+NULL
+
+
+-- !query
+select normalize('hello', 'not_a_form')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkRuntimeException
+{
+ "errorClass" : "INVALID_PARAMETER_VALUE.NORMALIZE_FORM",
+ "sqlState" : "22023",
+ "messageParameters" : {
+ "form" : "'not_a_form'",
+ "functionName" : "`normalize`",
+ "parameter" : "`form`"
+ }
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
index 3e0c8425c54e..00b6c926b92b 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
@@ -419,6 +419,16 @@ class StringFunctionsSuite extends SharedSparkSession {
// scalastyle:on
}
+ test("string normalize") {
+ // scalastyle:off
+ val df = Seq("\uFB01").toDF("s")
+ checkAnswer(df.select(normalize($"s")), Row("\uFB01"))
+ checkAnswer(df.select(normalize($"s", lit("NFKC"))), Row("fi"))
+ checkAnswer(df.selectExpr("normalize(s, 'NFKC')"), Row("fi"))
+ checkAnswer(df.select(normalize(lit(null), lit("NFC"))), Row(null))
+ // scalastyle:on
+ }
+
test("string translate") {
val df = Seq(("translate", "")).toDF("a", "b")
checkAnswer(df.select(translate($"a", "rnlt", "123")), Row("1a2s3ae"))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
