Hyukjin Kwon created SPARK-61321:
------------------------------------
Summary: DataFrame.take fails on a negative count, and transform
passes the callback a base DataFrame
Key: SPARK-61321
URL: https://issues.apache.org/jira/browse/SPARK-61321
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
DataFrame.take takes an unsigned count, so take(-1) fails with OverflowError in
argument conversion, while head, limit and tail send the count to the server,
which raises AnalysisException INVALID_LIMIT_LIKE_EXPRESSION.IS_NEGATIVE as
PySpark does. DataFrame.transform calls the function with a fresh base-class
DataFrame rather than the receiver, so for a DataFrame subclass MyDF(plan,
session).transform(lambda d: d.twice()) fails and transform(lambda d: d) loses
the subclass, where PySpark passes self. Repro: spark.range(3).take(-1);
MyDF(df._plan, spark).transform(lambda d: d.twice()).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
