Wangda Tan created YARN-10168:
---------------------------------
Summary: FS-CS Convert: Converter tool doesn't handle min/max
resource conversion correct
Key: YARN-10168
URL: https://issues.apache.org/jira/browse/YARN-10168
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Wangda Tan
Trying to understand logics of convert min and max resource from FS to CS, and
found some issues:
1)
In FSQueueConverter#emitMaximumCapacity
Existing logic in FS is to either specify a maximum percentage for queues
against cluster resources. Or, specify an absolute valued maximum resource.
In the existing FS2CS converter, when a percentage-based maximum resource is
specified, the converter takes a global resource from fs2cs CLI, and applies
percentages to that. It is not correct since the percentage-based value will
get lost, and in the future when cluster resources go up and down, the maximum
resource cannot be changed.
2)
The logic to deal with min/weight resource is also questionable:
The existing fs2cs tool, it takes precedence of percentage over
absoluteResource, and could set both to a queue config. See
FSQueueConverter.Capacity#toString
However, in CS, comparing to FS, the weights/min resource is quite different:
CS use the same queue.capacity to specify both percentage-based or
absolute-resource-based configs (Similar to how FS deal with maximum Resource).
The capacity defines guaranteed resource, which also impact fairshare of the
queue. (The more guaranteed resource a queue has, the larger "pie" the queue
can get if there's any additional available resource).
In FS, minResource defined the guaranteed resource, and weight defined how
much the pie can grow to.
So to me, in FS, we should pick and choose either weight or minResource to
generate CS.
3)
In FS, mix-use of absolute-resource configs (like min/maxResource), and
percentage-based (like weight) is allowed. But in CS, it is not allowed. The
reason is discussed on YARN-5881, and find [a]Should we support specifying a
mix of percentage ...
The existing fs2cs doesn't handle the issue, which could set mixed absolute
resource and percentage-based resources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
