srielau commented on code in PR #58549:
URL: https://github.com/apache/spark/pull/58549#discussion_r4110312606
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -157,7 +159,7 @@ def to_arrow_type(
arrow_type = pa.float64()
elif isinstance(dt, DecimalType):
arrow_type = pa.decimal128(dt.precision, dt.scale)
- elif isinstance(dt, StringType):
+ elif isinstance(dt, (StringType, CharType, VarcharType)):
Review Comment:
Fixed. Connect `createDataFrame` now runs Classic's UDT CHAR/VARCHAR
preflight on explicit and inferred schemas, and Connect `toArrow()` rejects the
same UDT-storage case before the shared Arrow mapper. Covered by Connect
regressions in `test_connect_dataframe_creation.py` and the mixin `toArrow` UDT
test inherited by Arrow parity.
##########
python/pyspark/sql/connect/udtf.py:
##########
@@ -163,13 +168,45 @@ def __init__(
if isinstance(returnType, str)
else returnType
)
+ # Regular UDTFs validate UDT storage while decorating, matching
Classic. PyArrow-native
Review Comment:
Applied.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala:
##########
@@ -456,9 +456,15 @@ case class PythonUDF(
// single lambda, and one more for each enclosing lambda when the UDF is
lifted out of a nested
// lambda (e.g. `transform(arr, i -> transform(i, x -> f(x)))` lifts `f`
to depth 2). Ignored
// for every non-element-wise eval type, where it stays at its default of
1.
- elementwiseNestingDepth: Int = 1)
+ elementwiseNestingDepth: Int = 1,
+ // The original CHAR/VARCHAR result type when write-side checks apply.
Generated equality
+ // includes this field, and canonicalized copies retain it, so checked and
unchecked UDFs
+ // remain distinct.
Review Comment:
Applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
