srielau commented on code in PR #58549:
URL: https://github.com/apache/spark/pull/58549#discussion_r4110304386
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -157,7 +159,7 @@ def to_arrow_type(
arrow_type = pa.float64()
elif isinstance(dt, DecimalType):
arrow_type = pa.decimal128(dt.precision, dt.scale)
- elif isinstance(dt, StringType):
+ elif isinstance(dt, (StringType, CharType, VarcharType)):
Review Comment:
Adding CHAR/VARCHAR to this shared mapper also silently enables the excluded
UDT-storage case in Connect. `connect.session.createDataFrame` leaves the UDT
wrapper intact, `LocalDataToArrowConversion` descends into `udt.sqlType()`, and
the server restores that wrapper; Connect `DataFrame.toArrow()` likewise calls
this mapper without Classic's `_has_char_varchar_in_udt` preflight. Please
reject these schemas at both Connect boundaries and add focused Connect
regressions so Connect matches Classic and the scope fence.
##########
python/pyspark/sql/connect/udtf.py:
##########
@@ -163,13 +168,45 @@ def __init__(
if isinstance(returnType, str)
else returnType
)
+ # Regular UDTFs validate UDT storage while decorating, matching
Classic. PyArrow-native
Review Comment:
```suggestion
# Regular UDTFs validate UDT storage when they are decorated,
matching Classic. PyArrow-native
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala:
##########
@@ -456,9 +456,15 @@ case class PythonUDF(
// single lambda, and one more for each enclosing lambda when the UDF is
lifted out of a nested
// lambda (e.g. `transform(arr, i -> transform(i, x -> f(x)))` lifts `f`
to depth 2). Ignored
// for every non-element-wise eval type, where it stays at its default of
1.
- elementwiseNestingDepth: Int = 1)
+ elementwiseNestingDepth: Int = 1,
+ // The original CHAR/VARCHAR result type when write-side checks apply.
Generated equality
+ // includes this field, and canonicalized copies retain it, so checked and
unchecked UDFs
+ // remain distinct.
Review Comment:
```suggestion
// This field contains the original CHAR/VARCHAR result type when
write-side checks apply.
// Generated equality includes this field, and canonicalized copies
retain it, so checked and
// unchecked UDFs remain distinct.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
