[
https://issues.apache.org/jira/browse/YARN-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129171#comment-16129171
]
Yufei Gu commented on YARN-6612:
--------------------------------
Thanks [~templedf] for the new patch. It makes more sense to me. Some comments:
- Would it be easier to understand if changing {{calculateShares()}} to
{{calculateShareUsages()}}? This applies to {{calculateMinShares()}} as well.
- What if weight is 0? The current patch throw a RuntimeException in method
{{calculateShares()}}. We also need unit tests for this.
- It doesn't need weight while calculating min share.
{code}
shares[i][1] = resourceInfo[i].getValue() /
(minShareInfo[i].getValue() * weights.getWeight(i));
{code}
For min share calculation, I guess we want to do it this way:
{code}
(usage/cluster)/(minShare/cluster)
{code}
which can be simplified to
{code}
usage/minShare
{code}
In addition, we need to handle zero-minShare, which needs unit tests.
- The parameter {{cluster}} in method {{compareShares()}} confuses me.
According to the last analysis, both min share and fair share in DRF should
consider cluster. The major difference between them are one uses weights, the
other use minShares.
- Right now, the code sort by fair share usage before comparison of min share.
Would it make more sense if we sort it by dominant min share which is
minShare/cluster?
{code}
// Calculate the min shares, then sort by fair share, and compare them
// by min share
calculateMinShares(usage1, minShare1, shares1, s1.getWeights());
calculateMinShares(usage2, minShare2, shares2, s2.getWeights());
sortShares(shares1, shares2);
res = compareShares(shares1, shares2, false);
{code}
- Could you add comments that first is the fair share and second is the min
share for {{float[][] share1}}?
- Parameter {{resource}} should be {{index}} in Java Doc of
{{ResourceWeights#getWeight(int)}}
- In {{ResourceWeights#getWeight(int)}}: {{sb.deleteCharAt(sb.length());}}
should be {{sb.deleteCharAt(sb.length() - 1);}}.
- ResourceInformation#toString is verbose (including unit, type, value, etc),
we don't want it in
ResourceWeights#ToString. Just output name is enough.
- typo: desceding in the method {{sortShares}}
- some test failures are related.
> Update fair scheduler policies to be aware of resource types
> ------------------------------------------------------------
>
> Key: YARN-6612
> URL: https://issues.apache.org/jira/browse/YARN-6612
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
> Affects Versions: YARN-3926
> Reporter: Daniel Templeton
> Assignee: Daniel Templeton
> Attachments: YARN-6612.YARN-3926.001.patch,
> YARN-6612.YARN-3926.002.patch, YARN-6612.YARN-3926.004.patch,
> YARN-6612.YARN-3926.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
