heyihong commented on code in PR #53935:
URL: https://github.com/apache/spark/pull/53935#discussion_r2739015258
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/ObservationManager.scala:
##########
@@ -54,7 +56,8 @@ private[sql] class ObservationManager(session: SparkSession) {
private[sql] def tryComplete(qe: QueryExecution): Unit = {
// Use lazy val to defer collecting the observed metrics until it is
needed so that tryComplete
// can finish faster (e.g., when the logical plan doesn't contain
CollectMetrics).
- lazy val lazyObservedMetrics = qe.observedMetrics
+ // Wrap in Try to capture potential failures when collecting metrics.
+ lazy val lazyObservedMetrics = Try(qe.observedMetrics)
Review Comment:
This is necessary because the collection of observed metrics is skipped (for
better efficiency) if the logical plan does not contain `CollectMetrics`, which
may occur in a significant portion of query executions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
