Yikun opened a new pull request, #36464: URL: https://github.com/apache/spark/pull/36464
### What changes were proposed in this pull request? Add groupby positional indexing support for Pandas on Spark. ### Why are the changes needed? Pandas supports Groupby positional indexing since v1.4.0 https://pandas.pydata.org/docs/whatsnew/v1.4.0.html#groupby-positional-indexing Before 1.4.0, `pdf.groupby("a").head(-2) == pdf.groupby("a").head(0)` return empty dataframe, after 1.4.0, it allows to specify positional ranges relative to the ends of each group. ```python df = pd.DataFrame([["g", "g0"], ["g", "g1"], ["g", "g2"], ["g", "g3"], ["h", "h0"], ["h", "h1"]], columns=["A", "B"]) df.groupby("A").head(-1) A B 0 g g0 1 g g1 2 g g2 4 h h0 ``` ### Does this PR introduce _any_ user-facing change? Yes, follow pandas 1.4+ behaviors ### How was this patch tested? - Added some more postional indexing test (-1, -100000). - `test_tail` and `test_head` pass with pandas 1.3.x and 1.4.x -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
