[
https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-5587:
--------------------------------
Attachment: YARN-5587-YARN-3926.008.patch
Thanks for the review [~leftnoteasy]!
bq. 1) Are Changes of ClusterNodeTrack necessary?
Nope. Fixed.
{quote}
2) ProfileCapability.toResource:
- Result of the if (resourceProfileMap != null) will be overwritten by if
(capability.getProfile..)
- Should order of the two if.. be exchanged?
{quote}
The logic is as follows -
- Use the profile to lookup the resources for the profile
- Apply any override specified in the profile capability override over the
profile resources
I think the logic is correct for that no?
{quote}
- Do we have a method to copy a Resource instead of using for... to clone
ResourceInformation
{quote}
Fixed.
{quote}
3) ResourceRequest / ProfileCapability / Resource
IIUC, one of getProfileCapability and getCapability will be selected based on
RM_RESOURCE_PROFILES_ENABLED. And
RMServerUtils#convertProfileToResourceCapability handles this logic to set
which is the effective capability. I have some questions/comments:
- How to handle cluster level profile update. For example, application request
a "small" container and waiting to be allocated. Before the container
allocated, admin updates capability of the "small" profile (can admin do things
like this?). In this case, should we allocate container with new configured
resource of "small" profile?
{quote}
This is currently not possible - the profiles are read on startup - but if we
ever would support it - the conversion happens in the ApplicationMasterService
as soon as the request is received but before it’s submitted to the scheduler.
So the time for the potential race condition is shorter but it would exist.
However, like I mentioned earlier, we don’t support this right now.
{quote}
3) getProfileCapability.getProfileCapabilityOverride get preferred even if
RM_RESOURCE_PROFILES_ENABLED is false, not sure if it is the best solution.
Here's my suggestion about API design:
It looks like ProfileCapability#getProfileCapabilityOverride will be only used
by application (admin won't set this field), if this is true, in existing YARN,
we have two fields for capability – ProfileCapability and Capability. Could we
treat the Capability as ProfileCapability#getProfileCapabilityOverride? This
approach has following benefits:
- Avoid putting application-required-only field to the common object
ProfileCapability which will be used by application and admin.
- Less option to application: application only need to set 2 fields
(capability, profile) instead of 3 (capability, profile, profile.override)
- It will also simplify some service/client logics such as RemoteRequestsTable.
{quote}
The logic is as follows - on the client side - all new clients will only use
ProfileCapability to specify the request, older clients will use Capability.
However at some point, I would like to deprecate Capability. Initially I had
written the code as you mentioned, but it becomes confusing to specify both the
ProfileCapability and the Capability in the APIs(as [~asuresh] has pointed out
above). It’s easier to construct one object and pass it around.
There are other benefits to using ProfileCapability - I suspect we’ll have to
add support for more shorthands, like resource profiles, in the future(for
resources such as ip’s, ports. etc) - these can be added to the
ProfileCapability without changing a whole bunch of interfaces like we’re
current doing. I actually would prefer to rename ProfileCapability to
ExtendedResourceCapability(but I can do that in a future JIRA).
The point about {quote} Less option to application: application only need to
set 2 fields (capability, profile) instead of 3 (capability, profile,
profile.override) {quote} shouldn’t matter too much. The older APIs to specify
just a resource are still preserved so clients can choose them if they desire.
They’re internally converted to a ProfileCapability before being sent to the RM.
bq. It will also simplify some service/client logics such as
RemoteRequestsTable.
I don’t think this will happen - for getMatchingRequests to work correctly,
ProfileCapability must be a first class concept in RemoteRequestsTable.
The TestAMRMClient failures are related to the patch. Once we've sorted out the
API discussion, I'll fix them.
> Add support for resource profiles
> ---------------------------------
>
> Key: YARN-5587
> URL: https://issues.apache.org/jira/browse/YARN-5587
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Reporter: Varun Vasudev
> Assignee: Varun Vasudev
> Labels: oct16-hard
> Attachments: YARN-5587-YARN-3926.001.patch,
> YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch,
> YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch,
> YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch,
> YARN-5587-YARN-3926.008.patch
>
>
> Add support for resource profiles on the RM side to allow users to use
> shorthands to specify resource requirements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
