mgaido91 commented on a change in pull request #23258:
[SPARK-23375][SQL][FOLLOWUP][TEST] Test Sort metrics while Sort is missing
URL: https://github.com/apache/spark/pull/23258#discussion_r244429920
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
##########
@@ -185,19 +190,105 @@ trait SQLMetricsTestUtils extends SQLTestUtils {
df: DataFrame,
expectedNumOfJobs: Int,
expectedMetrics: Map[Long, (String, Map[String, Any])]): Unit = {
- val optActualMetrics = getSparkPlanMetrics(df, expectedNumOfJobs,
expectedMetrics.keySet)
+ val expectedMetricsPredicates = expectedMetrics.mapValues { case
(nodeName, nodeMetrics) =>
+ (nodeName, nodeMetrics.mapValues(expectedMetricValue =>
+ (actualMetricValue: Any) => expectedMetricValue.toString ===
actualMetricValue)
+ )}
+ testSparkPlanMetricsWithPredicates(df, expectedNumOfJobs,
expectedMetricsPredicates)
+ }
+
+ /**
+ * Call `df.collect()` and verify if the collected metrics satisfy the
specified predicates.
+ * @param df `DataFrame` to run
+ * @param expectedNumOfJobs number of jobs that will run
+ * @param expectedMetricsPredicates the expected metrics predicates. The
format is
Review comment:
my point is: as of now, pattern matching is enough for what we need to check
and we do not have a use case when we actually need to parse the exact values.
Doing that, we can simplify this PR and reduce considerably the size of this
change. So I think we should go this way. If in the future we will need
something like you proposed here because we want to check the actual values,
then we can introduce all the methods you are suggesting here. But as of know
this can be skipped IMO.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
