[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598293#comment-16598293 ] Manikandan R commented on YARN-4175: [~asuresh] [~leftnoteasy] Can you please review the patch? > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING >Priority: Major > Attachments: YARN-4175.003.patch, YARN-4175.004.patch, > YARN-4175.1.patch, YARN-4175.2.patch > > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564057#comment-16564057 ] Manikandan R commented on YARN-4175: Thanks [~asuresh]. Updated earlier patch to do unit conversions for resource types if there is any difference in units between client side and server side (given units are not empty at both sides), rather than simply converting it to "Mi". For "memory" Mandatory resource, "Mi" would be assumed as unit if clients doesn't specify units and conversion would happen only if there is any difference between both values (given units are not empty at both sides). For "vcores" Mandatory resource, assuming unit is going to be empty always, left the code as it is. Also added a test case to cover this case as well. Please review .004 patch. > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING >Priority: Major > Attachments: YARN-4175.003.patch, YARN-4175.1.patch, YARN-4175.2.patch > > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560087#comment-16560087 ] Arun Suresh commented on YARN-4175: --- Thanks for taking this up.. yup makes sense . Do post your patch.. I'll try to take a look over the weekend. > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING >Priority: Major > Attachments: YARN-4175.003.patch, YARN-4175.1.patch, YARN-4175.2.patch > > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[
https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560054#comment-16560054
]
Manikandan R commented on YARN-4175:
Can you please review and share your comments?
{quote}Ideally, it has to be converted based on server side RM config
(Something similar to YARN-7159) . For example, If resource type 'resource1'
unit is 'Gi' configured at RM config and clients are passing values in
different units, it has to be converted to 'Gi' not to 'Mi'. Thoughts? I can
raise a separate JIRA to handle this separately based on comments.
{quote}
Does it makes sense? Can I start this new JIRA?
> Example of use YARN-1197
>
>
> Key: YARN-4175
> URL: https://issues.apache.org/jira/browse/YARN-4175
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: MENG DING
>Priority: Major
> Attachments: YARN-4175.003.patch, YARN-4175.1.patch, YARN-4175.2.patch
>
>
> Like YARN-2609, we need a example program to demonstrate how to use YARN-1197
> from end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542009#comment-16542009 ] Manikandan R commented on YARN-4175: I tried to use this patch to see if i can add more test cases for YARN-5590 with additional support from YARN-7242. I did the following to make this patch usable: # As [~asuresh] mentioned earlier, modified this patch to make use of UpdateContainer API's to update containers. Also ensured variables reflect the same in the code. However, retained the same shell arguments. 2. Had come to know that YARN-7242 doesn't take care of resource types units from shell arguments. It expects value of type long only and it works based on that. Made changes to pass resource types with units and the same would be considered for further process. But, it converts the value to "Mi" as of now. Ideally, it has to be converted based on server side RM config (Something similar to YARN-7159) . For example, If resource type 'resource1' unit is 'Gi' configured at RM config and clients are passing values in different units, it has to be converted to 'Gi' not to 'Mi'. Thoughts? I can raise a separate JIRA to handle this separately based on comments. 3. As intent is to add more test cases for YARN-5590, did the same to ensure containers can be updated using distributed shell both in junits and local pseudo set up. I can separate out the patch specific to YARN-5590 changes if needed. [~leftnoteasy] [~sunilg] [~asuresh] Please share your views. > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING >Priority: Major > Attachments: YARN-4175.1.patch, YARN-4175.2.patch > > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589683#comment-15589683 ] Wangda Tan commented on YARN-4175: -- Thanks [~asuresh] for helping this. I think [~mding] is on vacation now, please go ahead! > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING > Attachments: YARN-4175.1.patch, YARN-4175.2.patch > > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589653#comment-15589653 ] Arun Suresh commented on YARN-4175: --- [~mding], [~leftnoteasy], Mind if I take over this? I guess this should be updated to take YARN-5221 changes into consideration. > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING > Attachments: YARN-4175.1.patch, YARN-4175.2.patch > > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[
https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978444#comment-14978444
]
MENG DING commented on YARN-4175:
-
Correct a typo in the previous post. It should be {{app_id}} instead of
{{application_id}}
\\
* once the application has started, user can start a new client and specify the
*appmaster* option to set the client to the appmaster mode. Under this mode,
the client will talk directly with appmaster, and user can specify *app_id*,
*container_id*, *action*, *container_memory*, *container_vcores* options to
request container resizing. For example, to change a container resource, the
user can do:
{code}
hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -appmaster
-app_id= -container_id= -action=CHANGE_CONTAINER
-container_memory=2048 -container_vcores=1
{code}
> Example of use YARN-1197
>
>
> Key: YARN-4175
> URL: https://issues.apache.org/jira/browse/YARN-4175
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: MENG DING
> Attachments: YARN-4175.1.patch, YARN-4175.2.patch
>
>
> Like YARN-2609, we need a example program to demonstrate how to use YARN-1197
> from end-to-end.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945988#comment-14945988 ] MENG DING commented on YARN-4175: - Update on my testing result. Based on my tests of this feature against a 4 node cluster using the modified distributed shell app, the only critical issue I found is an NPE issue of resourcemanager when there is not enough headroom. The issue has been logged in YARN-4230. The only other minor issue I can think of is that some logging information can be improved, for which I will log a separate (low priority) issue. The tests I performed so far include: * Verify container resource increase/decrease when there are resources available, and no limits are exceeded. Verify container sizes are reported correctly on Web UI. * Verify container resource increase reservation when host doesn't have enough resource for the additional allocation. Verify resource reservation information on Web UI (Memory Reserved, Lasts Reservation, etc) * Verify that while an increase reservation is in place on a host, regular and increase allocation requests from other application will be skipped on this host. * Verify that an increase reservation will be fulfilled when enough resource is freed up on the host. * Verify that while increase reservation is in place for a container, a decrease request to the same container (with target resource <= original resource) will cancel the reservation. * Verify that pending resource increase request will not be processed when there is no headroom left (after applying patch from YARN-4230). * Verify that invalid resource increase/decrease request will throw exception in AMRMClient and distributed shell application master onError callback handler will be called. * Verify that resource monitoring is changed on NM after container increase/decrease is completed. * Verify that killing and restarting NM will recover increased/decreased containers if NM work preserving restart is enabled. * All tests are verified using both DefaultResourceCalculator and DominantResourceCalculator. Let me know if you have any comments or suggestions. > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING > Attachments: YARN-4175.1.patch > > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946015#comment-14946015 ] Wangda Tan commented on YARN-4175: -- Thanks [~mding] a lot! As an end-to-end test for basic functionalities, I think this should be enough, more tests could be done when downstream projects support container resizing, such as Spark/Slider. Will review pending patches you mentioned shortly. > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING > Attachments: YARN-4175.1.patch > > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944039#comment-14944039 ] MENG DING commented on YARN-4175: - I am using the example application to test the container increase/decrease function against a 4 node cluster. Will collect and report all problems when the tests are completed. Just a quick note in case someone also wants to do the test: * The application master IPC server now listens on a fixed port 8686. If multiple app masters are started on the same host with *-enable_ipc* option specified, there will be port conflicts, but YARN should be able to start new app attempts and try to launch app master on a different host. * If there are invalid container resource change request (e.g., target resource is smaller than original resource for increase), the AMRMClient will throw exception (i.e. InvalidResourceRequestException) at the allocate call, and current implementation of the distributed shell appmaster will exit, causing the entire application to exit. > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING > Attachments: YARN-4175.1.patch > > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938919#comment-14938919 ] MENG DING commented on YARN-4175: - Update on the progress of this ticket: The example will be based on the existing DistributedShell application. The idea is to add an RPC service to the DistributedShell application master, and also a client to issue requests to this service to increase/decrease container resources after the application is started. The patch is almost ready and under testing. Will post it for review soon. > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908676#comment-14908676 ] Wangda Tan commented on YARN-4175: -- Thanks for taking this [~mding], assigned to you! > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: MENG DING > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4175) Example of use YARN-1197
[ https://issues.apache.org/jira/browse/YARN-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908669#comment-14908669 ] MENG DING commented on YARN-4175: - Hi, [~leftnoteasy], if you are not working on this right now, I will be happy to take this one after YARN-1509 and YARN-1510 is done. I understand the urgency of the end-to-end test, and will treat this as the highest priority. > Example of use YARN-1197 > > > Key: YARN-4175 > URL: https://issues.apache.org/jira/browse/YARN-4175 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > > Like YARN-2609, we need a example program to demonstrate how to use YARN-1197 > from end-to-end. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
