[ 
https://issues.apache.org/jira/browse/SPARK-61409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61409:
-----------------------------------
    Labels: pull-request-available  (was: )

> Raise PySpark's errors for a list among the columns of repartition, a ternary 
> pow and an argument of Connect's _to_col that is not a column, and warn of a 
> Column key in getItem, in the Python client
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61409
>                 URL: https://issues.apache.org/jira/browse/SPARK-61409
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Four differences of the Python client from PySpark 4.3.0:
> 1. repartition(n, [cols]) and repartitionByRange(n, [cols]) unpack a single 
> list or tuple of columns. PySpark's give the tuple of *cols itself to 
> _to_cols, so a list among the columns is NOT_EXPECTED_TYPE ("Column or str", 
> cols), and repartitionByRange(n, []) is that error too, where the client 
> raised CANNOT_BE_EMPTY.
> 2. pow(col, 2, 3) and pow(2, col, 3) silently ignore the modulo. PySpark's 
> Column.__pow__ and __rpow__ take no modulo, so they raise TypeError 
> ("Column.__pow__() takes 2 positional arguments but 3 were given", and 
> Python's "unsupported operand type(s) for ** or pow()").
> 3. Column.getItem(<Column>) gives no FutureWarning; PySpark's warns "A column 
> as 'key' in getItem is deprecated as of Spark 3.0 ...", as the client's 
> getField already does for its name.
> 4. pyspark.sql.connect.functions.builtin._to_col is the lenient _to_col of 
> pyspark.sql.functions, which passes an argument that is not a Column or a str 
> on as it is. So pyspark.ml.functions.vector_to_array([1.0]) and 
> array_to_vector(1) build expressions, and from_protobuf(1, ...) and 
> to_protobuf([1], ...) raise a plain TypeError, where PySpark raises 
> NOT_EXPECTED_TYPE ("Column or str", col) for all of them.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to