Szilard Nemeth created YARN-9457:
------------------------------------
Summary: Integrate custom resource metrics better for FairScheduler
Key: YARN-9457
URL: https://issues.apache.org/jira/browse/YARN-9457
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Szilard Nemeth
YARN-8842 added
org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetricsForCustomResources.
This class stores all metrics data for custom resource types.
A field is there in QueueMetrics to hold an object of this class.
Similarly, YARN-9322 added FSQueueMetricsForCustomResources and added an object
of this class to FSQueueMetrics.
This jira is about to investigate how it is possible to integrate
QueueMetricsForCustomResources into QueueMetrics and
FSQueueMetricsForCustomResources into FSQueueMetrics.
The trick is that the Metrics annotation
(org.apache.hadoop.metrics2.annotation.Metric) is used to expose values on JMX.
We need to implement a mechanism where QueueMetrics / FSQueueMetrics classes do
contain a field of the custom resource values which is a map of resource names
as keys, and longs as values.
This way, we don't need the new classes (QueueMetricsForCustomResources and
FSQueueMetricsForCustomResources), the code could be much cleaner and
consistent.
The hardest part possibly is to find a way to expose metrics values from a map.
We obviously can't use the Metrics annotation so a mechanism is required to
expose the values on JMX.
For a quick search, I haven't found any way like this in the code
[~wilfreds]: Are you aware of any way to expose values like this?
Most probably, we need to check how the Metrics annotation is processed,
understand the whole flow and check what is the underlying mechanism of the
metrics propagation to the JMX interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
