viirya commented on pull request #31476:
URL: https://github.com/apache/spark/pull/31476#issuecomment-785558137
I share the direction to have custom merging behavior. So that being said, I
agree with the polished idea from the API perspective.
I don't have question about the task side. It is basically the same except
for the enum metric type.
My only question is how to delegate the aggregating logic to the aggregate
method. As we cannot access `CustomMetrics` during processing execution live
information. I think we need to change `SQLMetrics` to include the aggregate
method if it is `V2_CUSTOM` metrics type, e.g.
```scala
class SQLMetric(val metricType: String, initValue: Long = 0L,
customAggregateMethod: Option[Array[Long] -> String]) extends
AccumulatorV2[Long, Long] {
...
}
```
When we aggregate the metrics, for `V2_CUSTOM` metric type, we ask original
`SQLMetric` from `AccumulatorContext` and call the aggregate method.
Does it make sense?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
