grundprinzip commented on code in PR #39254:
URL: https://github.com/apache/spark/pull/39254#discussion_r1058092052
##########
python/pyspark/sql/connect/group.py:
##########
@@ -97,36 +97,46 @@ def agg(self, *exprs: Union[Column, Dict[str, str]]) ->
"DataFrame":
),
session=self._df._session,
)
- return res
agg.__doc__ = PySparkGroupedData.agg.__doc__
- def _map_cols_to_expression(self, fun: str, param: Union[Column, str]) ->
Sequence[Column]:
- return [
- scalar_function(fun, col(param)) if isinstance(param, str) else
param,
- ]
+ def _numeric_agg(self, function: str, cols: Sequence[str]) -> "DataFrame":
+ from pyspark.sql.connect.dataframe import DataFrame
+
+ assert isinstance(function, str) and function in ["min", "max", "avg",
"sum"]
+
+ assert isinstance(cols, list) and all(isinstance(c, str) for c in cols)
+
+ return DataFrame.withPlan(
Review Comment:
I see now why you do it that way, but I'm not convinced it's the best way.
The only delta I see that can't be handled reasonably is the case of empty
agg expressions where we know the function. And here I'm wondering if we should
just get the schema instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
