[ 
https://issues.apache.org/jira/browse/SPARK-61542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61542.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 571
[https://github.com/apache/spark-connect-rust/pull/571]

> Convert the counts of limit, offset, tail, head, take, repartition, 
> repartitionByRange, coalesce and repartitionById as protobuf's default 
> backend does, refusing a bool, as PySpark does
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61542
>                 URL: https://issues.apache.org/jira/browse/SPARK-61542
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Under protobuf's default backend (upb) on protobuf 7.34+, which a fresh 
> install of pyspark-client 4.3.0 gets, official PySpark converts a count that 
> it sets in a proto int field as upb does: a bool raises TypeError "Field 
> spark.connect.Limit.limit: Expected an int, got a boolean.", a value that is 
> not an int raises the TypeError of operator.index, and an int out of the 
> range of the field raises ValueError "Value out of range: N".
> The Python client took these counts through PyO3: a bool was sent as 1 or 0, 
> and the other errors were PyO3's, prefixed with the name of the argument 
> (OverflowError for a big int). This applies to DataFrame.limit, offset, tail, 
> head, take, repartition, repartitionByRange, coalesce and repartitionById.
> Each now runs PySpark's own checks first, in PySpark's order (the type 
> checks, `not numPartitions > 0` with VALUE_NOT_POSITIVE of its str, 
> CANNOT_BE_EMPTY), and then converts the count as upb does, with the full name 
> of its field. The helper proto_int is shared with dropna's thresh. A 
> partitionIdCol of repartitionById that is neither a Column nor a str now 
> raises PySpark's NOT_EXPECTED_TYPE of `col` (F._to_col) instead of a custom 
> TypeError.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to