skonto edited a comment on pull request #34039: URL: https://github.com/apache/spark/pull/34039#issuecomment-926435483
@BOOTMGR imho triggerExecution time looks a good candidate to be part of the metrics that should be exposed by Spark to the metrics system if needed automatically (what else is missing from the query reporting info?) without having to depend on the listener.I think long term would be nice to have that. It is fine to make the metrics system work as expected in the listener (this issue, it should be fixed), but the listener use case matches better for custom metrics (that is why I mentioned that). >I do not want to push it at every batch end (expensive sink), rather I would like to have them flushed along with other metrics via MetricsSystem. Makes sense if the sink is too expensive. However, in a pull model things are not much different, metrics will be sampled anyway at some interval eg. Prometheus, via some metrics exporter, independently of flushing. So the equivalent in terms of costs, for a push model would be to push at some appropriate rate. Anyway I was thinking out loud to see if there is more work to be done here than initially captured with this ticket. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
