Hyukjin Kwon created SPARK-61728:
------------------------------------

             Summary: Send a (value, DataType) argument of the native ML calls 
as the literal of that type, as PySpark does
                 Key: SPARK-61728
                 URL: https://issues.apache.org/jira/browse/SPARK-61728
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


serialize of pyspark.ml.connect.serialize sends a (value, DataType) pair, a 
tuple of two whose second item is a DataType, as LiteralExpression(value, 
dataType): the literal of that type. Its asserts raise a bare AssertionError of 
a value that is not of the type, such as ("x", IntegerType()), and of a type 
that no literal has (MapType, StructType, CharType, VarcharType, 
YearMonthIntervalType, a UDT), also with a None value.

The native ML classes of Spark Connect Rust raise the UNSUPPORTED_DATA_TYPE of 
a tuple for such an argument of their calls (ChiSquareTest.test, 
Correlation.corr, PowerIterationClustering.assignClusters, the members of a 
model), where official PySpark 4.3.0 sends the literal for the server to reject 
or raises its AssertionError.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to