[
https://issues.apache.org/jira/browse/YARN-10168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046405#comment-17046405
]
Peter Bacsko edited comment on YARN-10168 at 2/27/20 9:33 AM:
--
[~leftnoteasy] we really have to talk about this.
_"In the existing FS2CS converter, when a percentage-based maximum resource is
specified, the converter takes a global resource from fs2cs CLI, and applies
percentages to that. It is not correct since the percentage-based value will
get lost, and in the future when cluster resources go up and down, the maximum
resource cannot be changed."_
That's true, but you can't define a vector of percentages to CS at the moment.
That's why YARN-9936 was created, but unfortunately it hasn't been finished
yet. So you can have only a single percentage. How do you deal with that if the
input mem/vcore percentages are different?
_"In FS, minResource defined the guaranteed resource, and weight defined how
much the pie can grow to._
_So to me, in FS, we should pick and choose either weight or minResource to
generate CS."_
{{}} is optional in FS. You don't always have it. The only thing
that is mandatory is the weight. That's why weight was used as a starting
point.
_"In FS, mix-use of absolute-resource configs (like min/maxResource), and
percentage-based (like weight) is allowed. But in CS, it is not allowed."_
That's weird. I was under the impression that for static queue configs, you can
mix capacity and absolute resource. In this case, the verification of sum(caps)
== 100.0 is skipped. So is this assumption false?
was (Author: pbacsko):
[~leftnoteasy] we really have to talk about this.
_"In the existing FS2CS converter, when a percentage-based maximum resource is
specified, the converter takes a global resource from fs2cs CLI, and applies
percentages to that. It is not correct since the percentage-based value will
get lost, and in the future when cluster resources go up and down, the maximum
resource cannot be changed."_
That's true, but you can't define a vector of percentages to CS at the moment.
That's why YARN-9936 was created, but unfortunately it hasn't been finished
yet. So you can have only a single percentage. How do you deal with that if the
input mem/vcore percentages are different?
_"In FS, minResource defined the guaranteed resource, and weight defined how
much the pie can grow to._
_So to me, in FS, we should pick and choose either weight or minResource to
generate CS."_
{{}} is optional in FS. You don't always have it. The only thing
that is mandatory is the weight. That's why weight was used as a starting
point.
_"In FS, mix-use of absolute-resource configs (like min/maxResource), and
percentage-based (like weight) is allowed. But in CS, it is not allowed."_
That's weird. I was under the impression that for static queue configs, you can
mix capacity and absolute resource. In this case, the verification of sum(caps)
== 100.0 is skipped. So is this assumption false?
> FS-CS Convert: Converter tool doesn't handle min/max resource conversion
> correct
>
>
> Key: YARN-10168
> URL: https://issues.apache.org/jira/browse/YARN-10168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Priority: Blocker
>
> Trying to understand logics of convert min and max resource from FS to CS,
> and found some issues:
> 1)
> In FSQueueConverter#emitMaximumCapacity
> Existing logic in FS is to either specify a maximum percentage for queues
> against cluster resources. Or, specify an absolute valued maximum resource.
> In the existing FS2CS converter, when a percentage-based maximum resource is
> specified, the converter takes a global resource from fs2cs CLI, and applies
> percentages to that. It is not correct since the percentage-based value will
> get lost, and in the future when cluster resources go up and down, the
> maximum resource cannot be changed.
> 2)
> The logic to deal with min/weight resource is also questionable:
> The existing fs2cs tool, it takes precedence of percentage over
> absoluteResource, and could set both to a queue config. See
> FSQueueConverter.Capacity#toString
> However, in CS, comparing to FS, the weights/min resource is quite different:
> CS use the same queue.capacity to specify both percentage-based or
> absolute-resource-based configs (Similar to how FS deal with maximum
> Resource).
> The capacity defines guaranteed resource, which also impact fairshare of the
> queue. (The more guaranteed resource a queue has, the larger "pie" the queue
> can get if there's any additional available resource).
> In FS, minResource defined the guaranteed resource, and weight defined how
> much the pie can grow to.
> So to me, in FS,