[
https://issues.apache.org/jira/browse/SPARK-61024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61024.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 226
[https://github.com/apache/spark-connect-rust/pull/226]
> Raise the errors of PySpark for the arguments of DataFrame methods in the
> Python client
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-61024
> URL: https://issues.apache.org/jira/browse/SPARK-61024
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, DataFrame methods check their arguments on the client. They raise
> a PySparkTypeError of {{NOT_EXPECTED_TYPE}}, with the name, the expected type
> and the type of the argument, for: {{select}}, {{drop}}, {{repartition}} and
> {{groupBy}} of a value that is neither a Column nor a string (nor an ordinal
> for {{groupBy}}); {{sort}} and {{orderBy}} of one that is not a Column, an
> int or a string; {{toDF}} of a non-string; {{sample}} of arguments that are
> not a bool, a float or an int; {{hint}} of a name that is not a string;
> {{dropDuplicates}} and {{dropDuplicatesWithinWatermark}} of a subset that is
> not a list or tuple of strings; {{withColumn}} of a value that is not a
> Column; and {{withColumns}} and {{withColumnsRenamed}} of a value that is not
> a dict. {{hint}} raises {{INVALID_ITEM_FOR_CONTAINER}} for a parameter that
> is not a string, a float, an int, a Column or a list of them.
> {{repartition}}, {{repartitionByRange}} and {{coalesce}} raise
> {{VALUE_NOT_POSITIVE}} for a number of partitions that is not positive.
> {{sort}}, {{repartitionByRange}} and {{agg}} raise {{CANNOT_BE_EMPTY}}
> without columns.
> The Rust-backed Python client raises a TypeError for the types, such as
> {{argument 'col2': 'int' object is not an instance of 'str'}}, whose
> getCondition() is not NOT_EXPECTED_TYPE. It accepts the parameters of
> {{hint}} of other types, sends the numbers of partitions that are not
> positive to the server, which fails, and sorts by, repartitions by range by,
> and aggregates no columns. Raise the errors as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
