[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642148#comment-16642148
]
Íñigo Goiri commented on YARN-8827:
-----------------------------------
Thanks [~asuresh] for the patch.
I think we can fix the findbugs, checkstyles and so on from Yetus.
Other minor comments:
* I think that ContainersMonitorImpl#run is single threaded and all the
accesses to {{perAppResourceUtilization}} are made here, no need for the
thread-safe comment; nothing is here.
* In ContainersMonitorImpl, we are doing the change of units twice (#825), can
we refactor that into a proper ResourceUtilization and use it in both places?
* ContainersMonitorImpl#1106, what does it mean?
* TestResourceUtilizationAggregator#line66 fits in one single line. (There are
a few more places like #171, #174, etc).
* TestResourceUtilizationAggregator for values like 8192, you can do 8*GB as
the rest of the class.
* Can we do a more informed wait in TestResourceUtilizationAggregator instead
of sleep(1000)?
* mkmap() and e() are an OK pattern but I would probably call the second one
mkmapentry and add javadoc for the two explaining the purpose.
* Can we use an static import for {{Assert.assertEquals()}}?
* The code in {{testResourceUtilizationAggregation}} is a little repetitive.
Given that this class is only used for this, I would make the NMs part of the
class and trigger registrations and so on as part of the class. I would also
add a couple more comments throughout explaining how the apps are submitting
requests, etc. Specially the numbers after #184.
* TestResourceUtilizationAggregator#184 can it be made calculated from some
assignment we track?
> Plumb per app, per user and per queue resource utilization from the NM to RM
> ----------------------------------------------------------------------------
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Arun Suresh
> Assignee: Arun Suresh
> Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
