srielau commented on code in PR #58549:
URL: https://github.com/apache/spark/pull/58549#discussion_r4186250615
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/python/UserDefinedPythonFunction.scala:
##########
@@ -63,6 +64,11 @@ case class UserDefinedPythonFunction(
bufferType: DataType = null) {
def builder(e: Seq[Expression]): Expression = {
+ if (CharVarcharUtils.hasCharVarchar(dataType) ||
+ (bufferType != null && CharVarcharUtils.hasCharVarchar(bufferType))) {
Review Comment:
The split is on the PR head now (`e08ece68087`, rebased to current master).
The previous fix landed on `SPARK-59275` rather than this PR branch
(`serge-rielau_data/SPARK-59275`), so the 2026-10-05 review still saw the
combined check. Buffer-only failures report `Python UDAF buffer schemas` and
`bufferType.catalogString`; when both are constrained, the return type is
reported first.
##########
python/pyspark/sql/udf.py:
##########
@@ -213,12 +214,16 @@ def __init__(
)
self.evalType = evalType
self.deterministic = deterministic
+ if isinstance(returnType, DataType):
+ UserDefinedFunction._check_return_type(returnType, evalType)
Review Comment:
Fixed in e08ece68087. Classic and Connect constructors now run only
`_check_no_char_varchar`. Arrow/Pandas eval-type conversion stays on the
`returnType` property. CHAR/VARCHAR is still rejected at construction; a STRING
grouped-map Pandas UDF constructs and fails only when `returnType` is consumed.
##########
python/pyspark/sql/tests/test_types.py:
##########
@@ -659,6 +660,48 @@ def test_string_type_simple_string(self):
self.assertEqual(StringType("UTF8_LCASE").simpleString(), "string
collate UTF8_LCASE")
self.assertEqual(StringType("UNICODE").simpleString(), "string collate
UNICODE")
+ def test_check_no_char_varchar(self):
+ from pyspark.testing.objects import ExamplePointUDT
+
+ _check_no_char_varchar(StringType(), "Python UDF return types")
+ _check_no_char_varchar(ArrayType(StringType()), "Python UDF return
types")
+
+ cases = [
+ CharType(3),
+ VarcharType(8),
+ ArrayType(CharType(5)),
+ MapType(VarcharType(4), StringType()),
Review Comment:
Fixed in e08ece68087. `_check_no_char_varchar` now includes maps with a
STRING key and CHAR/VARCHAR in the value (including nested array values), plus
a `map<string, varchar(n)>` UDF return-type case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
