Hyukjin Kwon created SPARK-61243:
------------------------------------
Summary: Rust GroupedData sum/avg/min/max/mean ignore the numeric
defaults and the NOT_NUMERIC_COLUMNS check
Key: SPARK-61243
URL: https://issues.apache.org/jira/browse/SPARK-61243
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
GroupedData::sum, avg, min, max and mean in the Rust client mapped the given
column names straight to aggregate expressions. With no columns, sum(vec![])
built an aggregate with no expressions and returned only the grouping columns.
A non-numeric column went to the server unchecked. PySpark aggregates every
numeric column when no column is given and raises NOT_NUMERIC_COLUMNS for a
column that is not numeric. Repro: df.group_by(k).sum(vec![]) on a frame with
int, long, double and string columns returns only k; PySpark returns k, sum(i),
sum(l), sum(d).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
