HyukjinKwon commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561461000
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g.
75%)</li>
+ * <li>count_distinct</li>
Review comment:
I am not sure. You can all have what you want via `agg`. This is just a
shortcut to show basic stats, which was inspired by pandas' one, see
https://github.com/apache/spark/commit/e1a172c201d68406faa53b113518b10c879f1ff6.
We should of course avoid showing every stats here.
You can easily do `df.agg("col" -> "approx_count_distinct", ...)` easily for
other stats. Adding `count_distinct` here makes it ambiguous what to add in
`DataFrame.describe` and `DataFrame.summary`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
