LucaCanali commented on issue #24132: [SPARK-27189][CORE] Add executor-level memory usage metrics to the metrics system URL: https://github.com/apache/spark/pull/24132#issuecomment-477333935 Thank you @squito for taking time to review this. I agree with your questions and proposals for an improved implemetation. I'd just like to add to the thread the general idea/motivation of why I believe adding this type of metrics is useful. We use Spark metrics to feed a dashboard for performance monitoring of Spark applications. I have an example of how this works for us in https://db-blog.web.cern.ch/blog/luca-canali/2019-02-performance-dashboard-apache-spark Memory/executor metrics as I added them while testing this, appear to be a useful addition to the performance dashbord. In particular, I like very much the details into Spark memory usage exposed by the storage memory and execution memory metrics. As you mention, some memory-related metrics are available also from other sources, notably metrics defined in org.apache.spark.metrics.source.JvmSource. However, JvmSource is optional, so we could think that there is not really duplication of logged info: one could use the metrics introduced here instead of activating JvmSource, for example.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
