seancxmao commented on a change in pull request #23258:
[SPARK-23375][SQL][FOLLOWUP][TEST] Test Sort metrics while Sort is missing
URL: https://github.com/apache/spark/pull/23258#discussion_r244002544
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
##########
@@ -194,10 +194,26 @@ class SQLMetricsSuite extends SparkFunSuite with
SQLMetricsTestUtils with Shared
}
test("Sort metrics") {
- // Assume the execution plan is
- // WholeStageCodegen(nodeId = 0, Range(nodeId = 2) -> Sort(nodeId = 1))
- val ds = spark.range(10).sort('id)
- testSparkPlanMetrics(ds.toDF(), 2, Map.empty)
+ // Assume the execution plan with node id is
+ // Sort(nodeId = 0)
+ // Exchange(nodeId = 1)
+ // Project(nodeId = 2)
+ // LocalTableScan(nodeId = 3)
+ // Because of SPARK-25267, ConvertToLocalRelation is disabled in the test
cases of sql/core,
+ // so Project here is not collapsed into LocalTableScan.
+ val df = Seq(1, 3, 2).toDF("id").sort('id)
+ val metrics = getSparkPlanMetrics(df, 2, Set(0))
+ val sortMetrics = metrics.get.get(0).get
+ // Check node 0 is Sort node
+ val operatorName = sortMetrics._1
+ assert(operatorName == "Sort")
+ // Check metrics values
+ val sortTimeStr = sortMetrics._2.get("sort time total (min, med,
max)").get.toString
+ timingMetricStats(sortTimeStr).foreach { case (sortTime, _) =>
assert(sortTime >= 0) }
Review comment:
Yes, I'm checking all 4 stats, including sum, min, med and max. And sum
could also be 0. I ran the same query with spark-shell locally. below's the
screenshot.
<img width="302" alt="screen shot 2018-12-26 at 10 19 07 pm"
src="https://user-images.githubusercontent.com/12194089/50448637-4d991f80-095d-11e9-85ce-b2ee90d04e38.png";>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/12194089/50448637-4d991f80-095d-11e9-85ce-b2ee90d04e38.png"](https://user-images.githubusercontent.com/12194089/50448637-4d991f80-095d-11e9-85ce-b2ee90d04e38.png"){kind=link}