squito commented on a change in pull request #22874: [SPARK-25865][CORE] Add GC
information to ExecutorMetrics
URL: https://github.com/apache/spark/pull/22874#discussion_r243059130
##########
File path:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala
##########
@@ -99,6 +100,61 @@ case object ProcessTreeMetrics extends ExecutorMetricType {
}
}
+case object GarbageCollectionMetrics extends ExecutorMetricType {
+ import GC_TYPE._
+ override val names = Seq(
+ "MinorGCCount",
+ "MinorGCTime",
+ "MajorGCCount",
+ "MajorGCTime"
+ )
+ override private[spark] def getMetricValues(memoryManager: MemoryManager):
Array[Long] = {
+ var allMetrics = GCMetrics(0, 0, 0, 0)
+ ManagementFactory.getGarbageCollectorMXBeans.asScala.foreach { mxBean =>
+ val metrics = mxBean.getName match {
+ case `copy` | `psScavenge` | `parNew` | `g1Young` =>
Review comment:
this is a good point. I don't think the GarbageCollectorMXBeans (or
anything else in the JVM) define a mapping to major & minor GC. So we could
either:
a) manually update this mapping ourselves, as new GC methods come around.
also easy enough to make this configurable at runtime, so we don't have to
cover all cases, but I think we'd still try to do as much as possible
b) don't map them into major / minor at all, and just report the metrics
prefixed with the collector name. This is a little tricky, as then the number
of metrics is determined at runtime, based on what collectors are active in the
given JVM. But that should be manageable (it would wreck havoc if you had
inconsistent JVMs across executors, though).
do you have thoughts on what might be more useful to the end user?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
