Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5073#discussion_r26632524
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -752,6 +752,55 @@ class DataFrame private[sql](
}
/**
+ * Compute numerical statistics for given columns of this [[DataFrame]]:
+ * count, mean (avg), stddev (standard deviation), min, max.
+ * Each row of the resulting [[DataFrame]] contains column with
statistic name
+ * and columns with statistic results for each given column.
+ * If no columns are given then computes for all numerical columns.
+ *
+ * {{{
+ * df.describe("age", "height")
+ *
+ * // summary age height
+ * // count 10.0 10.0
+ * // mean 53.3 178.05
+ * // stddev 11.6 15.7
+ * // min 18.0 163.0
+ * // max 92.0 192.0
+ * }}}
+ */
+ @scala.annotation.varargs
+ def describe(cols: String*): DataFrame = {
+
+ def aggCol(name: String = "") = s"'$name' as summary"
+ val statistics = List[(String, Expression => Expression)](
+ "count" -> (expr => Count(expr)),
+ "mean" -> (expr => Average(expr)),
+ "stddev" -> (expr => Sqrt(Subtract(Average(Multiply(expr, expr)),
+ Multiply(Average(expr),
Average(expr))))),
+ "min" -> (expr => Min(expr)),
+ "max" -> (expr => Max(expr)))
+
+ val numCols = if (cols.isEmpty) numericColumns.map(_.prettyString)
else cols
+
+ // union all statistics starting from empty one
+ var description = selectExpr(aggCol()::numCols.toList:_*).limit(0)
--- End diff --
this query plan mignt result in going over the data multiple times. I think
an easy way to do this is to just run agg once, and pass it a set of
expressions. Then we can collect the value, take it out, and then create a
dataframe using a local sequence.
something like the following (probably not going to compile)
```scala
val aggCols = if (cols.isEmpty) numericColumns.map(_.prettyString) else cols
val aggExprs = aggCols.flatMap { col =>
Count(col) ::
Average(col) ::
Stddev(col) ::
Min(col) ::
Max(col) :: Nil
}
val result = agg(aggExprs).collect()
// pivot the result here to create a dataframe
...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
