Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19344#discussion_r141520163
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/TPCDSQueryBenchmark.scala
---
@@ -66,25 +68,23 @@ object TPCDSQueryBenchmark extends Logging {
classLoader = Thread.currentThread().getContextClassLoader)
// This is an indirect hack to estimate the size of each query's
input by traversing the
- // logical plan and adding up the sizes of all tables that appear in
the plan. Note that this
- // currently doesn't take WITH subqueries into account which might
lead to fairly inaccurate
- // per-row processing time for those cases.
- val queryRelations = scala.collection.mutable.HashSet[String]()
- spark.sql(queryString).queryExecution.logical.map {
- case UnresolvedRelation(t: TableIdentifier) =>
- queryRelations.add(t.table)
- case lp: LogicalPlan =>
- lp.expressions.foreach { _ foreach {
- case subquery: SubqueryExpression =>
- subquery.plan.foreach {
- case UnresolvedRelation(t: TableIdentifier) =>
- queryRelations.add(t.table)
- case _ =>
- }
- case _ =>
- }
+ // logical plan and adding up the sizes of all tables that appear in
the plan.
+ val planToCheck =
mutable.Stack[LogicalPlan](spark.sql(queryString).queryExecution.logical)
--- End diff --
Why not using the plan that has been analyzed?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
