This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 906af781492a [SPARK-48578][SQL][FOLLOWUP] add UTF8 string validation
related functions
906af781492a is described below
commit 906af781492ab7bdd19a90bbbec7f8cf2e59b4c8
Author: Uros Bojanic <[email protected]>
AuthorDate: Thu Jun 27 10:55:25 2024 +0900
[SPARK-48578][SQL][FOLLOWUP] add UTF8 string validation related functions
### What changes were proposed in this pull request?
Fix the dataType for `make_valid`.
### Why are the changes needed?
Wrong dataType was returned originally.
### Does this PR introduce _any_ user-facing change?
Yes, make_valid will now work properly.
### How was this patch tested?
Existing tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47106 from uros-db/fix-make-valid.
Authored-by: Uros Bojanic <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../org/apache/spark/sql/catalyst/expressions/stringExpressions.scala | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
index 476b18fac310..ca7d5c0248e3 100755
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
@@ -768,8 +768,7 @@ case class IsValidUTF8(input: Expression) extends
RuntimeReplaceable with Implic
case class MakeValidUTF8(input: Expression) extends RuntimeReplaceable with
ImplicitCastInputTypes
with UnaryLike[Expression] with NullIntolerant {
- override lazy val replacement: Expression = Invoke(
- input, "makeValid", SQLConf.get.defaultStringType)
+ override lazy val replacement: Expression = Invoke(input, "makeValid",
input.dataType)
override def inputTypes: Seq[AbstractDataType] = Seq(StringTypeAnyCollation)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
