xinrong-databricks commented on code in PR #36414:
URL: https://github.com/apache/spark/pull/36414#discussion_r863962657
##########
python/pyspark/pandas/generic.py:
##########
@@ -1312,11 +1326,20 @@ def sum(psser: "Series") -> Column:
return F.coalesce(F.sum(spark_column), SF.lit(0))
return self._reduce_for_stat_function(
- sum, name="sum", axis=axis, numeric_only=numeric_only,
min_count=min_count
+ sum,
+ name="sum",
+ axis=axis,
+ numeric_only=numeric_only,
+ min_count=min_count,
+ skipna=skipna,
)
def product(
- self, axis: Optional[Axis] = None, numeric_only: bool = None,
min_count: int = 0
+ self,
+ axis: Optional[Axis] = None,
+ skipna: bool = True,
Review Comment:
Good point!
One concern is maintaining pandas-on-Spark backward compatibility may
sacrifice API compatibility(parameter order) with pandas.
`skipna`exists in older pandas like 1.0.
Since Spark 3.4 is claimed to match pandas 1.4, shall we offer up
pandas-on-Spark backward compatibility in this case?
CC @HyukjinKwon @ueshin @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
