juliuszsompolski commented on code in PR #51091:
URL: https://github.com/apache/spark/pull/51091#discussion_r2156611908
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -69,6 +71,13 @@ case class MergeRowsExec(
copy(child = newChild)
}
+ override lazy val metrics: Map[String, SQLMetric] = Map(
+ "numTargetRowsCopied" -> SQLMetrics.createMetric(sparkContext,
+ "Number of target rows copied over because they did not match any
condition."),
+ "numTargetRowsUnmatched" -> SQLMetrics.createMetric(sparkContext,
+ "Number of target rows processed that do not match any condition. " +
+ "These will be dropped for delta-based merge and retained for
group-based merge."))
Review Comment:
The rows that are incrementing this metric are not copied - MergeRowsExec
drops them from output (return null), so in the end they are unused by the
WriteDeltaExec (not copied, not used by any of the conditions).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
