[
https://issues.apache.org/jira/browse/SPARK-61026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61026.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 227
[https://github.com/apache/spark-connect-rust/pull/227]
> Take the positional fraction and seed of sample, the ordinals of sort and
> groupBy, the Columns of drop and the list of selectExpr in the Python client
> as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61026
> URL: https://issues.apache.org/jira/browse/SPARK-61026
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, {{df.sample(withReplacement, fraction, seed)}} takes a fraction
> and a seed positionally: {{df.sample(0.5, 42)}} samples a fraction of 0.5
> with the seed 42. {{df.sort}}, {{df.orderBy}} and {{df.sortWithinPartitions}}
> take ordinals: {{df.sort(2)}} sorts by the second column and {{df.sort(-2)}}
> by it descending, and {{df.groupBy(2)}} groups by the second column.
> {{df.drop}} takes Columns as well as column names, such as {{df.drop(df.x)}},
> and {{df.selectExpr}} takes a list of expressions.
> The Rust-backed Python client takes the 42 of {{df.sample(0.5, 42)}} as the
> fraction, which fails on the server, raises a TypeError for the ordinals of
> {{sort}}, {{orderBy}}, {{sortWithinPartitions}} and {{groupBy}}, for the
> Columns of {{drop}} and for the list of {{selectExpr}}. Take them as PySpark
> does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
