[ https://issues.apache.org/jira/browse/YARN-10168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052333#comment-17052333 ]
Peter Bacsko edited comment on YARN-10168 at 3/5/20, 4:57 PM: -------------------------------------------------------------- After offline discussion with [~leftnoteasy], I arrived at the following conclusion: * drop support for {{<minResources>}} because it might be defined in absolute resources and it cannot be mixed with percentages in CS * drop support for {{<maxResources>}} for the very same reason * enhance the conveter to emit a WARNING when these settings are defined * Always set {{maximum-capacity = 100}} for every queue [~leftnoteasy] can you give green light for the changes above? was (Author: pbacsko): After offline discussion with [~leftnoteasy], I arrived at the following conclusion: * drop support for {{<minResources>}} because it might be defined in absolute resources and it cannot be mixed with percentages * drop support for {{<maxResources>}} for the very same reason * enhance the conveter to emit a WARNING when these settings are defined * Always set {{maximum-capacity = 100}} for every queue [~leftnoteasy] can you give green light for the changes above? > FS-CS Convert: Converter tool doesn't handle min/max resource conversion > correct > -------------------------------------------------------------------------------- > > Key: YARN-10168 > URL: https://issues.apache.org/jira/browse/YARN-10168 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Wangda Tan > Assignee: Peter Bacsko > Priority: Blocker > > Trying to understand logics of convert min and max resource from FS to CS, > and found some issues: > 1) > In FSQueueConverter#emitMaximumCapacity > Existing logic in FS is to either specify a maximum percentage for queues > against cluster resources. Or, specify an absolute valued maximum resource. > In the existing FS2CS converter, when a percentage-based maximum resource is > specified, the converter takes a global resource from fs2cs CLI, and applies > percentages to that. It is not correct since the percentage-based value will > get lost, and in the future when cluster resources go up and down, the > maximum resource cannot be changed. > 2) > The logic to deal with min/weight resource is also questionable: > The existing fs2cs tool, it takes precedence of percentage over > absoluteResource, and could set both to a queue config. See > FSQueueConverter.Capacity#toString > However, in CS, comparing to FS, the weights/min resource is quite different: > CS use the same queue.capacity to specify both percentage-based or > absolute-resource-based configs (Similar to how FS deal with maximum > Resource). > The capacity defines guaranteed resource, which also impact fairshare of the > queue. (The more guaranteed resource a queue has, the larger "pie" the queue > can get if there's any additional available resource). > In FS, minResource defined the guaranteed resource, and weight defined how > much the pie can grow to. > So to me, in FS, we should pick and choose either weight or minResource to > generate CS. > 3) > In FS, mix-use of absolute-resource configs (like min/maxResource), and > percentage-based (like weight) is allowed. But in CS, it is not allowed. The > reason is discussed on YARN-5881, and find [a]Should we support specifying a > mix of percentage ... > The existing fs2cs doesn't handle the issue, which could set mixed absolute > resource and percentage-based resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org