Hyukjin Kwon created SPARK-61024:
------------------------------------
Summary: Raise the NOT_EXPECTED_TYPE errors of PySpark for the
arguments of DataFrame methods in the Python client
Key: SPARK-61024
URL: https://issues.apache.org/jira/browse/SPARK-61024
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark, DataFrame methods check the types of their arguments on the client,
and raise a PySparkTypeError of {{NOT_EXPECTED_TYPE}} with the name, the
expected type and the type of the argument. Examples: {{select}}, {{drop}} and
{{repartition}} of a value that is neither a Column nor a string, {{toDF}} of a
non-string, {{sort}} of an {{ascending}} that is not a bool, an int or a list,
{{sample}} of a fraction that is not a float, {{hint}} of a name that is not a
string, {{dropDuplicates}} of a string subset, {{withColumns}} and
{{withColumnsRenamed}} of a value that is not a dict, {{cov}} and {{crosstab}}
of a column that is not a string, {{freqItems}} of {{cols}} and
{{approxQuantile}} of {{probabilities}} that are not a list or tuple, and
{{summary}} of a statistic that is not a string.
The Rust-backed Python client raises a TypeError for these, such as {{argument
'col2': 'int' object is not an instance of 'str'}}, whose getCondition() is not
NOT_EXPECTED_TYPE. Raise them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
