[
https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886273#comment-16886273
]
Manikandan R commented on YARN-6492:
------------------------------------
[~taklwu] Thanks for trying out the patch.
I will go through your findings and will incorporate the fixes if needed. Can
you please share more info like
# No. of nodes
# Labels info
# Node -> label mapping
# Type of labels - exclusive or non exclusive
[~eepayne] There are some improvements applied after .003.patch as well. So I
rebased using the same patch. Attaching .004.patch for your review. Currently,
it requires high level review on the approaches taken to compute
PartitionMetrics as it has been a while and the rest can be taken further as we
progress. Please refer earlier attachments PartitionQueueMetrics_*.txt to view
the JMX o/p.
Notes:
1. PartitionQueueMetrics extends QueueMetrics and act as holder for partition
queue metrics with just couple of methods to create objects.
2. Existing QueueMetrics class methods takes care of updating the partition
information too through PartitionQueueMetrics object
3. *Resource Vectors/Custom Resources Metrics:*
While we were working on this JIRA, based on suggestions, changes were made to
incorporate QueueMetrics even for resource vectors for ease of development. But
later, YARN-8842 has been created and committed as well. YARN-8842 patch
approach is different from the approach used in .004.patch.
.004.patch doesn't introduce any new class for this resource vectors metrics,
it just manages on its own with bit of extra logic. However, patch has to be
polished bit more to make it concrete. For example, As of now, it handles even
memory-mb and Vcores which can be removed as we progress. Below code in
{{QueueMetrics}} shows the behaviour:
{code:java}
for (ResourceInformation ri : res.getResources()) {
}
{code}
Ideally, we should traverse from 2nd index onwards. Likewise, We will need to
do some more minor improvements for sure.
4. Need to make it as robust patch - tighten the test cases, ensuring newly
added partition flow etc
5. Once High level design flow in patch has been concluded, can apply it even
for CSQueueMetrics as well.
> Generate queue metrics for each partition
> -----------------------------------------
>
> Key: YARN-6492
> URL: https://issues.apache.org/jira/browse/YARN-6492
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
> Reporter: Jonathan Hung
> Assignee: Manikandan R
> Priority: Major
> Attachments: PartitionQueueMetrics_default_partition.txt,
> PartitionQueueMetrics_x_partition.txt, PartitionQueueMetrics_y_partition.txt,
> YARN-6492.001.patch, YARN-6492.002.patch, YARN-6492.003.patch,
> partition_metrics.txt
>
>
> We are interested in having queue metrics for all partitions. Right now each
> queue has one QueueMetrics object which captures metrics either in default
> partition or across all partitions. (After YARN-6467 it will be in default
> partition)
> But having the partition metrics would be very useful.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
