srowen commented on a change in pull request #23258:
[SPARK-23375][SQL][FOLLOWUP][TEST] Test Sort metrics while Sort is missing
URL: https://github.com/apache/spark/pull/23258#discussion_r244533664
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
##########
@@ -198,6 +219,32 @@ trait SQLMetricsTestUtils extends SQLTestUtils {
}
}
}
+
+ /**
+ * Call `df.collect()` and verify if the collected metrics satisfy the
specified predicates.
+ * @param df `DataFrame` to run
+ * @param expectedNumOfJobs number of jobs that will run
+ * @param expectedMetricsPredicates the expected metrics predicates. The
format is
+ * `nodeId -> (operatorName, metric name -> metric value predicate)`.
+ */
+ protected def testSparkPlanMetricsWithPredicates(
+ df: DataFrame,
+ expectedNumOfJobs: Int,
+ expectedMetricsPredicates: Map[Long, (String, Map[String, Any =>
Boolean])]): Unit = {
+ val optActualMetrics =
+ getSparkPlanMetrics(df, expectedNumOfJobs,
expectedMetricsPredicates.keySet)
+ optActualMetrics.foreach { actualMetrics =>
+ assert(expectedMetricsPredicates.keySet === actualMetrics.keySet)
+ for (nodeId <- expectedMetricsPredicates.keySet) {
Review comment:
It might be a little cleaner to iterate over (key, value) pairs here and
below rather than iterate over keys then get values:
```
for ((nodeId, (expectedNodeName, expectedMetricsPredicatesMap) <-
expectedMetricsPredicates) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
