[
https://issues.apache.org/jira/browse/SPARK-61542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61542:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Convert the counts of limit, offset, tail, head, take, repartition,
> repartitionByRange, coalesce and repartitionById as protobuf's default
> backend does, refusing a bool, as PySpark does
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61542
> URL: https://issues.apache.org/jira/browse/SPARK-61542
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Under protobuf's default backend (upb) on protobuf 7.34+, which a fresh
> install of pyspark-client 4.3.0 gets, official PySpark converts a count that
> it sets in a proto int field as upb does: a bool raises TypeError "Field
> spark.connect.Limit.limit: Expected an int, got a boolean.", a value that is
> not an int raises the TypeError of operator.index, and an int out of the
> range of the field raises ValueError "Value out of range: N".
> The Python client took these counts through PyO3: a bool was sent as 1 or 0,
> and the other errors were PyO3's, prefixed with the name of the argument
> (OverflowError for a big int). This applies to DataFrame.limit, offset, tail,
> head, take, repartition, repartitionByRange, coalesce and repartitionById.
> Each now runs PySpark's own checks first, in PySpark's order (the type
> checks, `not numPartitions > 0` with VALUE_NOT_POSITIVE of its str,
> CANNOT_BE_EMPTY), and then converts the count as upb does, with the full name
> of its field. The helper proto_int is shared with dropna's thresh. A
> partitionIdCol of repartitionById that is neither a Column nor a str now
> raises PySpark's NOT_EXPECTED_TYPE of `col` (F._to_col) instead of a custom
> TypeError.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]