seancxmao commented on a change in pull request #23258:
[SPARK-23375][SQL][FOLLOWUP][TEST] Test Sort metrics while Sort is missing
URL: https://github.com/apache/spark/pull/23258#discussion_r244442573
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
##########
@@ -185,19 +190,105 @@ trait SQLMetricsTestUtils extends SQLTestUtils {
df: DataFrame,
expectedNumOfJobs: Int,
expectedMetrics: Map[Long, (String, Map[String, Any])]): Unit = {
- val optActualMetrics = getSparkPlanMetrics(df, expectedNumOfJobs,
expectedMetrics.keySet)
+ val expectedMetricsPredicates = expectedMetrics.mapValues { case
(nodeName, nodeMetrics) =>
+ (nodeName, nodeMetrics.mapValues(expectedMetricValue =>
+ (actualMetricValue: Any) => expectedMetricValue.toString ===
actualMetricValue)
+ )}
+ testSparkPlanMetricsWithPredicates(df, expectedNumOfJobs,
expectedMetricsPredicates)
+ }
+
+ /**
+ * Call `df.collect()` and verify if the collected metrics satisfy the
specified predicates.
+ * @param df `DataFrame` to run
+ * @param expectedNumOfJobs number of jobs that will run
+ * @param expectedMetricsPredicates the expected metrics predicates. The
format is
Review comment:
@mgaido91 I agree. Thanks for your detailed and clear explanation. Checking
metric values do make things unnecessarily complex.
@srowen As @mgaido91 said, currently it is not necessary to check metric
values, pattern matching is enough, and we could eliminate these methods. As
for code duplication, methods here are not duplicate with code in `Utils`.
`Utils` provides a bunch of methods to do conversion between string and bytes,
bytes there are of type `Long`. However bytes in metric values are of type
`Float`, e.g. `96.2 MB`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
