[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5587: Fix Version/s: (was: 2.10.0) > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Major > Labels: oct16-hard > Fix For: 3.1.0 > > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch, > YARN-5587-YARN-3926.012.patch, YARN-5587-YARN-3926.013.patch, > YARN-5587-YARN-3926.014.patch, YARN-5587-YARN-3926.015.patch, > YARN-5587.branch-3.0.001.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5587: Fix Version/s: 2.10.0 > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Major > Labels: oct16-hard > Fix For: 3.1.0, 2.10.0 > > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch, > YARN-5587-YARN-3926.012.patch, YARN-5587-YARN-3926.013.patch, > YARN-5587-YARN-3926.014.patch, YARN-5587-YARN-3926.015.patch, > YARN-5587.branch-3.0.001.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5587: --- Attachment: YARN-5587.branch-3.0.001.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Fix For: 3.1.0 > > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch, > YARN-5587-YARN-3926.012.patch, YARN-5587-YARN-3926.013.patch, > YARN-5587-YARN-3926.014.patch, YARN-5587-YARN-3926.015.patch, > YARN-5587.branch-3.0.001.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5587: - Fix Version/s: (was: YARN-3926) 3.1.0 > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Fix For: 3.1.0 > > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch, > YARN-5587-YARN-3926.012.patch, YARN-5587-YARN-3926.013.patch, > YARN-5587-YARN-3926.014.patch, YARN-5587-YARN-3926.015.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5587: -- Fix Version/s: YARN-3926 > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Fix For: YARN-3926 > > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch, > YARN-5587-YARN-3926.012.patch, YARN-5587-YARN-3926.013.patch, > YARN-5587-YARN-3926.014.patch, YARN-5587-YARN-3926.015.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.015.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch, > YARN-5587-YARN-3926.012.patch, YARN-5587-YARN-3926.013.patch, > YARN-5587-YARN-3926.014.patch, YARN-5587-YARN-3926.015.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.014.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch, > YARN-5587-YARN-3926.012.patch, YARN-5587-YARN-3926.013.patch, > YARN-5587-YARN-3926.014.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.013.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch, > YARN-5587-YARN-3926.012.patch, YARN-5587-YARN-3926.013.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.012.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch, > YARN-5587-YARN-3926.012.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.011.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.010.patch Uploaded new patch with findbugs and unit tests fixed. > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.009.patch Uploaded a new patch to fix the failing test cases and the javadoc issues. [~asuresh], [~leftnoteasy] - can you you please review? Thanks! > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[
https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-5587:
Attachment: YARN-5587-YARN-3926.008.patch
Thanks for the review [~leftnoteasy]!
bq. 1) Are Changes of ClusterNodeTrack necessary?
Nope. Fixed.
{quote}
2) ProfileCapability.toResource:
- Result of the if (resourceProfileMap != null) will be overwritten by if
(capability.getProfile..)
- Should order of the two if.. be exchanged?
{quote}
The logic is as follows -
- Use the profile to lookup the resources for the profile
- Apply any override specified in the profile capability override over the
profile resources
I think the logic is correct for that no?
{quote}
- Do we have a method to copy a Resource instead of using for... to clone
ResourceInformation
{quote}
Fixed.
{quote}
3) ResourceRequest / ProfileCapability / Resource
IIUC, one of getProfileCapability and getCapability will be selected based on
RM_RESOURCE_PROFILES_ENABLED. And
RMServerUtils#convertProfileToResourceCapability handles this logic to set
which is the effective capability. I have some questions/comments:
- How to handle cluster level profile update. For example, application request
a "small" container and waiting to be allocated. Before the container
allocated, admin updates capability of the "small" profile (can admin do things
like this?). In this case, should we allocate container with new configured
resource of "small" profile?
{quote}
This is currently not possible - the profiles are read on startup - but if we
ever would support it - the conversion happens in the ApplicationMasterService
as soon as the request is received but before it’s submitted to the scheduler.
So the time for the potential race condition is shorter but it would exist.
However, like I mentioned earlier, we don’t support this right now.
{quote}
3) getProfileCapability.getProfileCapabilityOverride get preferred even if
RM_RESOURCE_PROFILES_ENABLED is false, not sure if it is the best solution.
Here's my suggestion about API design:
It looks like ProfileCapability#getProfileCapabilityOverride will be only used
by application (admin won't set this field), if this is true, in existing YARN,
we have two fields for capability – ProfileCapability and Capability. Could we
treat the Capability as ProfileCapability#getProfileCapabilityOverride? This
approach has following benefits:
- Avoid putting application-required-only field to the common object
ProfileCapability which will be used by application and admin.
- Less option to application: application only need to set 2 fields
(capability, profile) instead of 3 (capability, profile, profile.override)
- It will also simplify some service/client logics such as RemoteRequestsTable.
{quote}
The logic is as follows - on the client side - all new clients will only use
ProfileCapability to specify the request, older clients will use Capability.
However at some point, I would like to deprecate Capability. Initially I had
written the code as you mentioned, but it becomes confusing to specify both the
ProfileCapability and the Capability in the APIs(as [~asuresh] has pointed out
above). It’s easier to construct one object and pass it around.
There are other benefits to using ProfileCapability - I suspect we’ll have to
add support for more shorthands, like resource profiles, in the future(for
resources such as ip’s, ports. etc) - these can be added to the
ProfileCapability without changing a whole bunch of interfaces like we’re
current doing. I actually would prefer to rename ProfileCapability to
ExtendedResourceCapability(but I can do that in a future JIRA).
The point about {quote} Less option to application: application only need to
set 2 fields (capability, profile) instead of 3 (capability, profile,
profile.override) {quote} shouldn’t matter too much. The older APIs to specify
just a resource are still preserved so clients can choose them if they desire.
They’re internally converted to a ProfileCapability before being sent to the RM.
bq. It will also simplify some service/client logics such as
RemoteRequestsTable.
I don’t think this will happen - for getMatchingRequests to work correctly,
ProfileCapability must be a first class concept in RemoteRequestsTable.
The TestAMRMClient failures are related to the patch. Once we've sorted out the
API discussion, I'll fix them.
> Add support for resource profiles
> -
>
> Key: YARN-5587
> URL: https://issues.apache.org/jira/browse/YARN-5587
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Labels: oct16-hard
> Attachments: YARN-5587-YARN-3926.001.patch,
> YARN-5587-YARN-3926.002.patch,
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.007.patch Uploaded a new patch to fix the findbugs errors. [~asuresh] - question about RemoteRequestsTable - is it meant to be public to AMs or can we change APIs for it without any issues? > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[
https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-5587:
Attachment: YARN-5587-YARN-3926.006.patch
Thanks for the review [~asuresh]! I've uploaded a new patch to address your
comments.
{quote}
We create a special ProfileCapability with profile name NONE where user
only specifies the capability (Resource). This would be equivalent to our old
Resource
We replace the capability(Resource) argument in both the ContainerRequest
and ResourceRequest with ProfileCapability. Maybe for backward compatibility,
we have 1 constructor that takes Resource and 1 that takes ProfileCapability.
We then convert a Resource to ProfileCapability(NONE, Resource) in the
AMRMClient.
{quote}
Fixed. I've left the old Resource interfaces for backwards compatibility but
internally everything is getting converted to a ProfileCapability. Instead of
NONE, I'm setting the profile to "default". On the RM side, the default profile
is used and overriden with the Resource values.
{quote}
In the rest of the AMRMClient codebase, including the RemoteRequestsTable}
}we just use {{ProfileCapability
{quote}
Fixed.
{quote}
Hmmm... Actually, I think we have to deal with one other issue in the
AMRMClient:
Once a Container is returned by the RM, the AMRMClient uses the
getMatchingRequests() api to retrieve matching ContainerRequests and remove it.
This implies that we need to have an equivalence relationship from a Resource
to a ProfileCapability, else the matching won't work. Thus I think we might
have to normalize all "named" ProfileCapabilities to a either a Resource or a
NONE ProfileCapability (with the override being the complete resource)
Either that, or we have to ensure that these new requests are tagged with
unique 'allocationRequestIds', for which we already have a separate
getMatchingRequests that matches ContainerRequests to Containers only based on
the allocationRequestId.
{quote}
I missed this - thanks for pointing it out. I added a function to convert
profiles back to the Resource values to make the functionality work.
> Add support for resource profiles
> -
>
> Key: YARN-5587
> URL: https://issues.apache.org/jira/browse/YARN-5587
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Labels: oct16-hard
> Attachments: YARN-5587-YARN-3926.001.patch,
> YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch,
> YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch,
> YARN-5587-YARN-3926.006.patch
>
>
> Add support for resource profiles on the RM side to allow users to use
> shorthands to specify resource requirements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-5587: --- Labels: oct16-hard (was: ) > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.005.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.004.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: (was: YARN-5587-YARN-3926.003.patch) > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.003.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.003.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.003.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.002.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.001.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-5587-YARN-3926.001.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
