[jira] [Commented] (YARN-10168) FS-CS Convert: Converter tool doesn't handle min/max resource conversion correct
[
https://issues.apache.org/jira/browse/YARN-10168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052333#comment-17052333
]
Peter Bacsko commented on YARN-10168:
-
After offline discussion with [~leftnoteasy], I arrived at the following
conclusion:
* drop support for {{}} because it might be defined in absolute
resources and it cannot be mixed with percentages
* drop support for {{}} for the very same reason
* enhance the conveter to emit a WARNING when these settings are defined
* Always set {{maximum-capacity = 100}} for every queue
[~leftnoteasy] can you give green light for the changes above?
> FS-CS Convert: Converter tool doesn't handle min/max resource conversion
> correct
>
>
> Key: YARN-10168
> URL: https://issues.apache.org/jira/browse/YARN-10168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Peter Bacsko
>Priority: Blocker
>
> Trying to understand logics of convert min and max resource from FS to CS,
> and found some issues:
> 1)
> In FSQueueConverter#emitMaximumCapacity
> Existing logic in FS is to either specify a maximum percentage for queues
> against cluster resources. Or, specify an absolute valued maximum resource.
> In the existing FS2CS converter, when a percentage-based maximum resource is
> specified, the converter takes a global resource from fs2cs CLI, and applies
> percentages to that. It is not correct since the percentage-based value will
> get lost, and in the future when cluster resources go up and down, the
> maximum resource cannot be changed.
> 2)
> The logic to deal with min/weight resource is also questionable:
> The existing fs2cs tool, it takes precedence of percentage over
> absoluteResource, and could set both to a queue config. See
> FSQueueConverter.Capacity#toString
> However, in CS, comparing to FS, the weights/min resource is quite different:
> CS use the same queue.capacity to specify both percentage-based or
> absolute-resource-based configs (Similar to how FS deal with maximum
> Resource).
> The capacity defines guaranteed resource, which also impact fairshare of the
> queue. (The more guaranteed resource a queue has, the larger "pie" the queue
> can get if there's any additional available resource).
> In FS, minResource defined the guaranteed resource, and weight defined how
> much the pie can grow to.
> So to me, in FS, we should pick and choose either weight or minResource to
> generate CS.
> 3)
> In FS, mix-use of absolute-resource configs (like min/maxResource), and
> percentage-based (like weight) is allowed. But in CS, it is not allowed. The
> reason is discussed on YARN-5881, and find [a]Should we support specifying a
> mix of percentage ...
> The existing fs2cs doesn't handle the issue, which could set mixed absolute
> resource and percentage-based resources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10168) FS-CS Convert: Converter tool doesn't handle min/max resource conversion correct
[ https://issues.apache.org/jira/browse/YARN-10168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046736#comment-17046736 ] Wangda Tan commented on YARN-10168: --- [~pbacsko], what you mentioned are all make sense to me. I think we should only support weight convert to capacity for now (and set max to 100). It will have reliable behavior. This is the only blocker issue we need to fix. For min/maxResource we should push to another JIRA. > FS-CS Convert: Converter tool doesn't handle min/max resource conversion > correct > > > Key: YARN-10168 > URL: https://issues.apache.org/jira/browse/YARN-10168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Priority: Blocker > > Trying to understand logics of convert min and max resource from FS to CS, > and found some issues: > 1) > In FSQueueConverter#emitMaximumCapacity > Existing logic in FS is to either specify a maximum percentage for queues > against cluster resources. Or, specify an absolute valued maximum resource. > In the existing FS2CS converter, when a percentage-based maximum resource is > specified, the converter takes a global resource from fs2cs CLI, and applies > percentages to that. It is not correct since the percentage-based value will > get lost, and in the future when cluster resources go up and down, the > maximum resource cannot be changed. > 2) > The logic to deal with min/weight resource is also questionable: > The existing fs2cs tool, it takes precedence of percentage over > absoluteResource, and could set both to a queue config. See > FSQueueConverter.Capacity#toString > However, in CS, comparing to FS, the weights/min resource is quite different: > CS use the same queue.capacity to specify both percentage-based or > absolute-resource-based configs (Similar to how FS deal with maximum > Resource). > The capacity defines guaranteed resource, which also impact fairshare of the > queue. (The more guaranteed resource a queue has, the larger "pie" the queue > can get if there's any additional available resource). > In FS, minResource defined the guaranteed resource, and weight defined how > much the pie can grow to. > So to me, in FS, we should pick and choose either weight or minResource to > generate CS. > 3) > In FS, mix-use of absolute-resource configs (like min/maxResource), and > percentage-based (like weight) is allowed. But in CS, it is not allowed. The > reason is discussed on YARN-5881, and find [a]Should we support specifying a > mix of percentage ... > The existing fs2cs doesn't handle the issue, which could set mixed absolute > resource and percentage-based resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10168) FS-CS Convert: Converter tool doesn't handle min/max resource conversion correct
[ https://issues.apache.org/jira/browse/YARN-10168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046406#comment-17046406 ] Peter Bacsko commented on YARN-10168: - BTW is this really a blocker for the converter? > FS-CS Convert: Converter tool doesn't handle min/max resource conversion > correct > > > Key: YARN-10168 > URL: https://issues.apache.org/jira/browse/YARN-10168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Priority: Blocker > > Trying to understand logics of convert min and max resource from FS to CS, > and found some issues: > 1) > In FSQueueConverter#emitMaximumCapacity > Existing logic in FS is to either specify a maximum percentage for queues > against cluster resources. Or, specify an absolute valued maximum resource. > In the existing FS2CS converter, when a percentage-based maximum resource is > specified, the converter takes a global resource from fs2cs CLI, and applies > percentages to that. It is not correct since the percentage-based value will > get lost, and in the future when cluster resources go up and down, the > maximum resource cannot be changed. > 2) > The logic to deal with min/weight resource is also questionable: > The existing fs2cs tool, it takes precedence of percentage over > absoluteResource, and could set both to a queue config. See > FSQueueConverter.Capacity#toString > However, in CS, comparing to FS, the weights/min resource is quite different: > CS use the same queue.capacity to specify both percentage-based or > absolute-resource-based configs (Similar to how FS deal with maximum > Resource). > The capacity defines guaranteed resource, which also impact fairshare of the > queue. (The more guaranteed resource a queue has, the larger "pie" the queue > can get if there's any additional available resource). > In FS, minResource defined the guaranteed resource, and weight defined how > much the pie can grow to. > So to me, in FS, we should pick and choose either weight or minResource to > generate CS. > 3) > In FS, mix-use of absolute-resource configs (like min/maxResource), and > percentage-based (like weight) is allowed. But in CS, it is not allowed. The > reason is discussed on YARN-5881, and find [a]Should we support specifying a > mix of percentage ... > The existing fs2cs doesn't handle the issue, which could set mixed absolute > resource and percentage-based resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10168) FS-CS Convert: Converter tool doesn't handle min/max resource conversion correct
[
https://issues.apache.org/jira/browse/YARN-10168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046405#comment-17046405
]
Peter Bacsko commented on YARN-10168:
-
[~leftnoteasy] we really have to talk about this.
_"In the existing FS2CS converter, when a percentage-based maximum resource is
specified, the converter takes a global resource from fs2cs CLI, and applies
percentages to that. It is not correct since the percentage-based value will
get lost, and in the future when cluster resources go up and down, the maximum
resource cannot be changed."_
That's true, but you can't define a vector of percentages to CS at the moment.
That's why YARN-9936 was created, but unfortunately it hasn't been finished
yet. So you can have only a single percentage. How do you deal with that if the
input mem/vcore percentages are different?
_"In FS, minResource defined the guaranteed resource, and weight defined how
much the pie can grow to._
_So to me, in FS, we should pick and choose either weight or minResource to
generate CS."_
{{}} is optional in FS. You don't always have it. The only thing
that is mandatory is the weight. That's why weight was used as a starting
point.
_"In FS, mix-use of absolute-resource configs (like min/maxResource), and
percentage-based (like weight) is allowed. But in CS, it is not allowed."_
That's weird. I was under the impression that for static queue configs, you can
mix capacity and absolute resource. In this case, the verification of sum(caps)
== 100.0 is skipped. So is this assumption false?
> FS-CS Convert: Converter tool doesn't handle min/max resource conversion
> correct
>
>
> Key: YARN-10168
> URL: https://issues.apache.org/jira/browse/YARN-10168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Priority: Blocker
>
> Trying to understand logics of convert min and max resource from FS to CS,
> and found some issues:
> 1)
> In FSQueueConverter#emitMaximumCapacity
> Existing logic in FS is to either specify a maximum percentage for queues
> against cluster resources. Or, specify an absolute valued maximum resource.
> In the existing FS2CS converter, when a percentage-based maximum resource is
> specified, the converter takes a global resource from fs2cs CLI, and applies
> percentages to that. It is not correct since the percentage-based value will
> get lost, and in the future when cluster resources go up and down, the
> maximum resource cannot be changed.
> 2)
> The logic to deal with min/weight resource is also questionable:
> The existing fs2cs tool, it takes precedence of percentage over
> absoluteResource, and could set both to a queue config. See
> FSQueueConverter.Capacity#toString
> However, in CS, comparing to FS, the weights/min resource is quite different:
> CS use the same queue.capacity to specify both percentage-based or
> absolute-resource-based configs (Similar to how FS deal with maximum
> Resource).
> The capacity defines guaranteed resource, which also impact fairshare of the
> queue. (The more guaranteed resource a queue has, the larger "pie" the queue
> can get if there's any additional available resource).
> In FS, minResource defined the guaranteed resource, and weight defined how
> much the pie can grow to.
> So to me, in FS, we should pick and choose either weight or minResource to
> generate CS.
> 3)
> In FS, mix-use of absolute-resource configs (like min/maxResource), and
> percentage-based (like weight) is allowed. But in CS, it is not allowed. The
> reason is discussed on YARN-5881, and find [a]Should we support specifying a
> mix of percentage ...
> The existing fs2cs doesn't handle the issue, which could set mixed absolute
> resource and percentage-based resources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
