dongjoon-hyun commented on code in PR #47516: URL: https://github.com/apache/spark/pull/47516#discussion_r1697997183
########## sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala: ########## @@ -96,9 +96,12 @@ class SQLMetric( def +=(v: Long): Unit = add(v) - // _value may be uninitialized, in many cases being -1. We should not expose it to the user - // and instead return 0. - override def value: Long = if (isZero) 0 else _value + // We use -1 as initial value of the SIZE and TIMIMG accumulators (0 is a valid metric value). + // We need to return it as it is so that the SQL UI can filter out the invalid accumulator + // values in `SQLMetrics.stringValue` when calculating min, max, etc. + // However, users accessing the values in the physical plan programmatically still gets -1. They + // may use `SQLMetric.isZero` before consuming this value. + override def value: Long = _value Review Comment: Well, actually, this is a direct reverting of #39311 . - In this case, we need a review from the author because he proposed this as a kind of regression for accumulator values. - In addition, could you add a new unit test case for your case? IIUC, this PR only seems to update the existing test cases. cc @cloud-fan and @viirya and @HyukjinKwon from #39311 , too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
