[
https://issues.apache.org/jira/browse/SPARK-61321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61321:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> DataFrame.take fails on a negative count, and transform passes the callback a
> base DataFrame
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-61321
> URL: https://issues.apache.org/jira/browse/SPARK-61321
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> DataFrame.take takes an unsigned count, so take(-1) fails with OverflowError
> in argument conversion, while head, limit and tail send the count to the
> server, which raises AnalysisException
> INVALID_LIMIT_LIKE_EXPRESSION.IS_NEGATIVE as PySpark does.
> DataFrame.transform calls the function with a fresh base-class DataFrame
> rather than the receiver, so for a DataFrame subclass MyDF(plan,
> session).transform(lambda d: d.twice()) fails and transform(lambda d: d)
> loses the subclass, where PySpark passes self. Repro:
> spark.range(3).take(-1); MyDF(df._plan, spark).transform(lambda d: d.twice()).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
