Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/23224#discussion_r239837818
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
---
@@ -80,8 +80,10 @@ class SQLMetricsSuite extends SparkFunSuite with
SQLMetricsTestUtils with Shared
// Assume the execution plan is
// WholeStageCodegen(nodeId = 0, Range(nodeId = 2) -> Filter(nodeId =
1))
// TODO: update metrics in generated operators
- val ds = spark.range(10).filter('id < 5)
- testSparkPlanMetrics(ds.toDF(), 1, Map.empty)
+ val df = spark.range(10).filter('id < 5).toDF()
+ testSparkPlanMetrics(df, 1, Map.empty, true)
+
df.queryExecution.executedPlan.find(_.isInstanceOf[WholeStageCodegenExec])
+ .getOrElse(assert(false))
--- End diff --
Seems test `Sort metric` also has similar issue:
```scala
test("Sort metrics") {
// Assume the execution plan is
// WholeStageCodegen(nodeId = 0, Range(nodeId = 2) -> Sort(nodeId = 1))
val ds = spark.range(10).sort('id)
testSparkPlanMetrics(ds.toDF(), 2, Map.empty)
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
