zhengruifeng opened a new pull request, #36554: URL: https://github.com/apache/spark/pull/36554
### What changes were proposed in this pull request? Improve the numerical stability of skewness for cases with small `m2` and `m3` ### Why are the changes needed? the formulas to compute skewness were adjusted to be the same in https://github.com/apache/spark/pull/36549 but for cases with small `m2` (like constant sequances), the results are still different. That is because `pandas.core.nanops.nanskew` will zero out small `m2` and `m3` ``` m2 = _zero_out_fperr(m2) m3 = _zero_out_fperr(m3) ``` and return the final result as ``` result = 0 if m2 == 0 else result if count < 3: return np.nan ``` This PR is to introduce this logic into Pandas API on Spark. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? added UT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
