heyihong commented on code in PR #53935:
URL: https://github.com/apache/spark/pull/53935#discussion_r2739015258
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/ObservationManager.scala:
##########
@@ -54,7 +56,8 @@ private[sql] class ObservationManager(session: SparkSession) {
private[sql] def tryComplete(qe: QueryExecution): Unit = {
// Use lazy val to defer collecting the observed metrics until it is
needed so that tryComplete
// can finish faster (e.g., when the logical plan doesn't contain
CollectMetrics).
- lazy val lazyObservedMetrics = qe.observedMetrics
+ // Wrap in Try to capture potential failures when collecting metrics.
+ lazy val lazyObservedMetrics = Try(qe.observedMetrics)
Review Comment:
It is needed because observed metrics collection will be skipped if the
logical plan does not contain CollectMetrics, which may be the case for a
significant portion of query executions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
