[
https://issues.apache.org/jira/browse/YARN-10085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020041#comment-17020041
]
Adam Antal commented on YARN-10085:
-----------------------------------
Thanks for the patch [~pbacsko]!
It seems logically correct to me.
A question to have bigger context for me:
Could you please give some explanation why we have to explicitly set the
resource calculator class in {{FSYarnSiteConverter#convertSiteProperties}}? I
mean, if DRF is used: yes, it makes sense, but we could technically leave it
empty is {{drfUsed}} is false, couldn't we?
As a minor nit, I'd add a simple optimization in
{{FSConfigToCSConfigConverter#isDrfUsed}}:
If we have DRF in some leaf queue level we have to climb the whole queue
hierarchy just to find out that some leaf queue has that policy - so in some
cases it seems to be an expensive operation. Moreover, in {{#isDrfUsed}} we
might return true anyways if this line is true:
{code:java}
return DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)
{code}
So I assume we can do some minor performance improvement in that function by
reorganizing the calls (one of the cases could be moved top, and we immediately
return with true, if that is the case). Regarding what operation cost less you
may have more insight, so I'm also fine with climbing the queue hierarchy first
and then check the {{AllocationConfiguration}} (if that is the more expensive
one).
> FS-CS converter: remove mixed ordering policy check
> ---------------------------------------------------
>
> Key: YARN-10085
> URL: https://issues.apache.org/jira/browse/YARN-10085
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Peter Bacsko
> Assignee: Peter Bacsko
> Priority: Critical
> Attachments: YARN-10085-001.patch, YARN-10085-002.patch
>
>
> In the converter, this part is very strict and probably unnecessary:
> {noformat}
> // Validate ordering policy
> if (queueConverter.isDrfPolicyUsedOnQueueLevel()) {
> if (queueConverter.isFifoOrFairSharePolicyUsed()) {
> throw new ConversionException(
> "DRF ordering policy cannot be used together with fifo/fair");
> } else {
> capacitySchedulerConfig.set(
> CapacitySchedulerConfiguration.RESOURCE_CALCULATOR_CLASS,
> DominantResourceCalculator.class.getCanonicalName());
> }
> }
> {noformat}
> It's also misleading, because Fair policy can be used under DRF, so the error
> message is incorrect.
> Let's remove these checks and rewrite the converter in a way that it
> generates a valid config even if fair/drf is somehow mixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
