Hyukjin Kwon created SPARK-61043:
------------------------------------
Summary: Take the arguments of count and sum_distinct, and warn of
sumDistinct, in the Python client as PySpark does
Key: SPARK-61043
URL: https://issues.apache.org/jira/browse/SPARK-61043
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{functions.count(col)}} and
{{functions.sum_distinct(col)}} take a Column or a column name, and raise
{{NOT_EXPECTED_TYPE}} for anything else, as their {{_to_col}} does. {{count}}
of a star Column, as {{count(col("*"))}} or {{count(df["*"])}}, counts the rows
as {{count(1)}}. {{sumDistinct}}, deprecated, warns {{FutureWarning: Deprecated
in 3.4, use sum_distinct instead.}}
The Rust-backed Python client raises a {{TypeError}} for {{count(1)}}, takes
{{sum_distinct(1)}} as the sum of the distinct values of the literal {{1}},
sends {{count(col("*"))}} as the count of a star, and does not warn of
{{sumDistinct}}. Take them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
