spark git commit: [SPARK-15792][SQL] Allows operator to change the verbosity in explain output
Repository: spark
Updated Branches:
refs/heads/branch-2.0 a5bec5b81 -> 57dd4efcd
[SPARK-15792][SQL] Allows operator to change the verbosity in explain output
## What changes were proposed in this pull request?
This PR allows customization of verbosity in explain output. After change,
`dataframe.explain()` and `dataframe.explain(true)` has different verbosity
output for physical plan.
Currently, this PR only enables verbosity string for operator
`HashAggregateExec` and `SortAggregateExec`. We will gradually enable verbosity
string for more operators in future.
**Less verbose mode:** dataframe.explain(extended = false)
`output=[count(a)#85L]` is **NOT** displayed for HashAggregate.
```
scala> Seq((1,2,3)).toDF("a", "b", "c").createTempView("df2")
scala> spark.sql("select count(a) from df2").explain()
== Physical Plan ==
*HashAggregate(key=[], functions=[count(1)])
+- Exchange SinglePartition
+- *HashAggregate(key=[], functions=[partial_count(1)])
+- LocalTableScan
```
**Verbose mode:** dataframe.explain(extended = true)
`output=[count(a)#85L]` is displayed for HashAggregate.
```
scala> spark.sql("select count(a) from df2").explain(true) //
"output=[count(a)#85L]" is added
...
== Physical Plan ==
*HashAggregate(key=[], functions=[count(1)], output=[count(a)#85L])
+- Exchange SinglePartition
+- *HashAggregate(key=[], functions=[partial_count(1)], output=[count#87L])
+- LocalTableScan
```
## How was this patch tested?
Manual test.
Author: Sean Zhong
Closes #13535 from clockfly/verbose_breakdown_2.
(cherry picked from commit 5f731d6859c4516941e5f90c99c966ef76268864)
Signed-off-by: Cheng Lian
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/57dd4efc
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/57dd4efc
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/57dd4efc
Branch: refs/heads/branch-2.0
Commit: 57dd4efcda9158646df41ea8d70754dc110ecd6f
Parents: a5bec5b
Author: Sean Zhong
Authored: Mon Jun 6 22:59:25 2016 -0700
Committer: Cheng Lian
Committed: Mon Jun 6 22:59:34 2016 -0700
--
.../sql/catalyst/expressions/Expression.scala | 4
.../spark/sql/catalyst/plans/QueryPlan.scala| 2 ++
.../spark/sql/catalyst/trees/TreeNode.scala | 23 +++-
.../spark/sql/execution/QueryExecution.scala| 14 +++-
.../sql/execution/WholeStageCodegenExec.scala | 6 +++--
.../execution/aggregate/HashAggregateExec.scala | 12 --
.../execution/aggregate/SortAggregateExec.scala | 12 --
7 files changed, 55 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/57dd4efc/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
index 2ec4621..efe592d 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
@@ -190,6 +190,10 @@ abstract class Expression extends TreeNode[Expression] {
case single => single :: Nil
}
+ // Marks this as final, Expression.verboseString should never be called, and
thus shouldn't be
+ // overridden by concrete classes.
+ final override def verboseString: String = simpleString
+
override def simpleString: String = toString
override def toString: String = prettyName + flatArguments.mkString("(", ",
", ")")
http://git-wip-us.apache.org/repos/asf/spark/blob/57dd4efc/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
index 19a66cf..cf34f4b 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
@@ -257,6 +257,8 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]]
extends TreeNode[PlanT
override def simpleString: String = statePrefix + super.simpleString
+ override def verboseString: String = simpleString
+
/**
* All the subqueries of current plan.
*/
http://git-wip-us.apache.org/repos/asf/spark/blob/57dd4efc/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
--
diff --git
a/sql/catalyst/src/main/scala/
spark git commit: [SPARK-15792][SQL] Allows operator to change the verbosity in explain output
Repository: spark
Updated Branches:
refs/heads/master 0e0904a2f -> 5f731d685
[SPARK-15792][SQL] Allows operator to change the verbosity in explain output
## What changes were proposed in this pull request?
This PR allows customization of verbosity in explain output. After change,
`dataframe.explain()` and `dataframe.explain(true)` has different verbosity
output for physical plan.
Currently, this PR only enables verbosity string for operator
`HashAggregateExec` and `SortAggregateExec`. We will gradually enable verbosity
string for more operators in future.
**Less verbose mode:** dataframe.explain(extended = false)
`output=[count(a)#85L]` is **NOT** displayed for HashAggregate.
```
scala> Seq((1,2,3)).toDF("a", "b", "c").createTempView("df2")
scala> spark.sql("select count(a) from df2").explain()
== Physical Plan ==
*HashAggregate(key=[], functions=[count(1)])
+- Exchange SinglePartition
+- *HashAggregate(key=[], functions=[partial_count(1)])
+- LocalTableScan
```
**Verbose mode:** dataframe.explain(extended = true)
`output=[count(a)#85L]` is displayed for HashAggregate.
```
scala> spark.sql("select count(a) from df2").explain(true) //
"output=[count(a)#85L]" is added
...
== Physical Plan ==
*HashAggregate(key=[], functions=[count(1)], output=[count(a)#85L])
+- Exchange SinglePartition
+- *HashAggregate(key=[], functions=[partial_count(1)], output=[count#87L])
+- LocalTableScan
```
## How was this patch tested?
Manual test.
Author: Sean Zhong
Closes #13535 from clockfly/verbose_breakdown_2.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5f731d68
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5f731d68
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5f731d68
Branch: refs/heads/master
Commit: 5f731d6859c4516941e5f90c99c966ef76268864
Parents: 0e0904a
Author: Sean Zhong
Authored: Mon Jun 6 22:59:25 2016 -0700
Committer: Cheng Lian
Committed: Mon Jun 6 22:59:25 2016 -0700
--
.../sql/catalyst/expressions/Expression.scala | 4
.../spark/sql/catalyst/plans/QueryPlan.scala| 2 ++
.../spark/sql/catalyst/trees/TreeNode.scala | 23 +++-
.../spark/sql/execution/QueryExecution.scala| 14 +++-
.../sql/execution/WholeStageCodegenExec.scala | 6 +++--
.../execution/aggregate/HashAggregateExec.scala | 12 --
.../execution/aggregate/SortAggregateExec.scala | 12 --
7 files changed, 55 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/5f731d68/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
index 2ec4621..efe592d 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
@@ -190,6 +190,10 @@ abstract class Expression extends TreeNode[Expression] {
case single => single :: Nil
}
+ // Marks this as final, Expression.verboseString should never be called, and
thus shouldn't be
+ // overridden by concrete classes.
+ final override def verboseString: String = simpleString
+
override def simpleString: String = toString
override def toString: String = prettyName + flatArguments.mkString("(", ",
", ")")
http://git-wip-us.apache.org/repos/asf/spark/blob/5f731d68/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
index 19a66cf..cf34f4b 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
@@ -257,6 +257,8 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]]
extends TreeNode[PlanT
override def simpleString: String = statePrefix + super.simpleString
+ override def verboseString: String = simpleString
+
/**
* All the subqueries of current plan.
*/
http://git-wip-us.apache.org/repos/asf/spark/blob/5f731d68/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/c
