vinodkc commented on code in PR #58233: URL: https://github.com/apache/spark/pull/58233#discussion_r3849032233
########## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Mode.scala: ########## Review Comment: > just curious how the collation affect the PandasMode which is only used in Pandas API on spark? `PandasMode` keys its count map on the original value, so under a non-binary collation 'b' and 'B' are counted separately and the mode can be wrong. The fix folds collation-equal entries at eval time (shared with `Mode`); binary types are unaffected. Collation reaches it when a ps object wraps a collated Spark DataFrame . pandas-on-Spark has no collation concept of its own. > and do we need a new test in PS side to check the code change? Not really, pandas-on-Spark has no collation support, and PS tests compare against pandas, which also has none. I tested it at the expression level in `CollationAggregationSuite`, calling it the same way SF.mode does. Can add a Python smoke test if you'd prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
