sririshindra commented on a change in pull request #28173:
[SPARK-31389][SQL][TESTS] Ensure all tests in SQLMetricsSuite run with both
codegen on and off
URL: https://github.com/apache/spark/pull/28173#discussion_r409610752
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
##########
@@ -214,14 +214,16 @@ trait SQLMetricsTestUtils extends SQLTestUtils {
protected def testSparkPlanMetrics(
df: DataFrame,
expectedNumOfJobs: Int,
- expectedMetrics: Map[Long, (String, Map[String, Any])]): Unit = {
+ expectedMetrics: Map[Long, (String, Map[String, Any])],
+ enableWholeStage: Boolean = false): Unit = {
val expectedMetricsPredicates = expectedMetrics.mapValues { case
(nodeName, nodeMetrics) =>
(nodeName, nodeMetrics.mapValues(expectedMetricValue =>
(actualMetricValue: Any) => {
actualMetricValue.toString.matches(expectedMetricValue.toString)
}))
}
- testSparkPlanMetricsWithPredicates(df, expectedNumOfJobs,
expectedMetricsPredicates)
+ testSparkPlanMetricsWithPredicates(df, expectedNumOfJobs,
expectedMetricsPredicates,
Review comment:
Interesting idea to make the 'df' fresh.
However we still have the problem of expectedMetricsPredicates argument
being different for most of the tests when codegen=true and codegen=false.
Since the arguments to the testSparkPlanMetricsWithPredicates method could be
different when codegen=true and codegen=false, we probably couldn’t avoid
calling the testSparkPlanMetrics method twice with the corresponding arguments.
I agree with you that we should minimize the code duplication. I just pushed
another commit to further refactor the code to minimize code duplication. Could
you please take a look and let me know what do you think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
