[jira] [Commented] (YARN-407) The requested AM memory is not checked against the maximum resource capacity in ResourceMgrDelegate and MRAppMaster
[ https://issues.apache.org/jira/browse/YARN-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581515#comment-13581515 ] Hitesh Shah commented on YARN-407: -- @Zhijie, does this seem like a dup of YARN-193? The requested AM memory is not checked against the maximum resource capacity in ResourceMgrDelegate and MRAppMaster --- Key: YARN-407 URL: https://issues.apache.org/jira/browse/YARN-407 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen For the MR jobs, the requested AM memory is at the risk of being larger than the maximum resource capacity. It is because the logic to check the requested AM memory against the maximum resource capacity is missing in ResourceMgrDelegate and MRAppMaster. In contrast, this logic exists in Client and ApplicationMaster of the distributed shell. Another potential risk is that the requested AM memory is likely to be rounded up when being normalized, therefore the after-normalization AM memory can be larger than the maximum resource capacity as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-407) The requested AM memory is not checked against the maximum resource capacity in ResourceMgrDelegate and MRAppMaster
[ https://issues.apache.org/jira/browse/YARN-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581534#comment-13581534 ] Zhijie Shen commented on YARN-407: -- @Hitesh, I've quickly gone through your patch for YARN-193. I agree the issue is duplicate. Let's close it. The requested AM memory is not checked against the maximum resource capacity in ResourceMgrDelegate and MRAppMaster --- Key: YARN-407 URL: https://issues.apache.org/jira/browse/YARN-407 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen For the MR jobs, the requested AM memory is at the risk of being larger than the maximum resource capacity. It is because the logic to check the requested AM memory against the maximum resource capacity is missing in ResourceMgrDelegate and MRAppMaster. In contrast, this logic exists in Client and ApplicationMaster of the distributed shell. Another potential risk is that the requested AM memory is likely to be rounded up when being normalized, therefore the after-normalization AM memory can be larger than the maximum resource capacity as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-407) The requested AM memory is not checked against the maximum resource capacity in ResourceMgrDelegate and MRAppMaster
[ https://issues.apache.org/jira/browse/YARN-407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved YARN-407. -- Resolution: Duplicate The requested AM memory is not checked against the maximum resource capacity in ResourceMgrDelegate and MRAppMaster --- Key: YARN-407 URL: https://issues.apache.org/jira/browse/YARN-407 Project: Hadoop YARN Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen For the MR jobs, the requested AM memory is at the risk of being larger than the maximum resource capacity. It is because the logic to check the requested AM memory against the maximum resource capacity is missing in ResourceMgrDelegate and MRAppMaster. In contrast, this logic exists in Client and ApplicationMaster of the distributed shell. Another potential risk is that the requested AM memory is likely to be rounded up when being normalized, therefore the after-normalization AM memory can be larger than the maximum resource capacity as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-193) Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits
[ https://issues.apache.org/jira/browse/YARN-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581553#comment-13581553 ] Zhijie Shen commented on YARN-193: -- Hi [~hitesh], I've one comment on the patch. I've found that in the patch, the normalization with through exception if the requested resource is larger than the configured max value. IMHO, it's better to normalize the requested resource to the multiple of the min value, which is also not larger than max value. For example min = 1024, max = 2560, resource = 2300. If only min value is considered, the resource will be normalized to 3072. Then, if max value is considered, the resource should be reduce to the max multiple of min value, but no larger than max value, i.e., 2048. Otherwise, if the exception is anyway to be thrown in normalization, Client and ApplicationMaster of distributed shell should be modified accordingly, because the request resource are reset to the max value there if it is larger than the max value. I'm feeling that it's better to have the consistent behavior. Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits - Key: YARN-193 URL: https://issues.apache.org/jira/browse/YARN-193 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.0.2-alpha, 3.0.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Attachments: MR-3796.1.patch, MR-3796.2.patch, MR-3796.3.patch, MR-3796.wip.patch, YARN-193.4.patch, YARN-193.5.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[ https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-365: --- Attachment: YARN-365.6.patch Each NM heartbeat should not generate an event for the Scheduler Key: YARN-365 URL: https://issues.apache.org/jira/browse/YARN-365 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager, scheduler Affects Versions: 0.23.5 Reporter: Siddharth Seth Assignee: Xuan Gong Attachments: Prototype2.txt, Prototype3.txt, YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch, YARN-365.5.patch, YARN-365.6.patch Follow up from YARN-275 https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-365) Each NM heartbeat should not generate an event for the Scheduler
[ https://issues.apache.org/jira/browse/YARN-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581701#comment-13581701 ] Siddharth Seth commented on YARN-365: - Xuan, Thanks for updating the patch. Comments on the latest patch. There's some formatting issues - exceeding the 80 width limit, spaces after commas, etc. Also, there's some formatting changes to code which is unrelated to the patch which should be avoided. - Don't think the RMNode internal counter for number of queued events should be exposed. Infact, it can be implemented as a boolean instead of an integer for now, which gets reset whenever the scheduler tries fetching the list of container updates. Additional interfaces can be introduced when this behaviour is changed in the future. - The nodeUpdateQueue should be cleared early, before sending out the NodeRemovedEvent - applies to StatusUpdateWhenHealthyTransition, DeactivateNodeTransition and ReconnectNodeTransition - getContainerInfoList can be renamed to getContainerUpdates - In the unit tests, TestRMNodeTransition.setup() can be simplified. Don't think the 'first' flag is required. Also, testExpiredContainer and testStatusChange need to be updated after the latest change to the patch. Each NM heartbeat should not generate an event for the Scheduler Key: YARN-365 URL: https://issues.apache.org/jira/browse/YARN-365 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager, scheduler Affects Versions: 0.23.5 Reporter: Siddharth Seth Assignee: Xuan Gong Attachments: Prototype2.txt, Prototype3.txt, YARN-365.1.patch, YARN-365.2.patch, YARN-365.3.patch, YARN-365.4.patch, YARN-365.5.patch, YARN-365.6.patch Follow up from YARN-275 https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-408) Capacity Scheduler delay scheduling should not be disabled by default
Mayank Bansal created YARN-408: -- Summary: Capacity Scheduler delay scheduling should not be disabled by default Key: YARN-408 URL: https://issues.apache.org/jira/browse/YARN-408 Project: Hadoop YARN Issue Type: Bug Reporter: Mayank Bansal Assignee: Mayank Bansal Priority: Minor Capacity Scheduler delay scheduling should not be disabled by default. Enabling it to number of nodes in one rack. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-408) Capacity Scheduler delay scheduling should not be disabled by default
[ https://issues.apache.org/jira/browse/YARN-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-408: --- Attachment: YARN-408-trunk.patch Attaching patch Thanks, Mayank Capacity Scheduler delay scheduling should not be disabled by default - Key: YARN-408 URL: https://issues.apache.org/jira/browse/YARN-408 Project: Hadoop YARN Issue Type: Bug Reporter: Mayank Bansal Assignee: Mayank Bansal Priority: Minor Attachments: YARN-408-trunk.patch Capacity Scheduler delay scheduling should not be disabled by default. Enabling it to number of nodes in one rack. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-408) Capacity Scheduler delay scheduling should not be disabled by default
[ https://issues.apache.org/jira/browse/YARN-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-408: --- Component/s: scheduler Affects Version/s: 3.0.0 2.0.3-alpha Capacity Scheduler delay scheduling should not be disabled by default - Key: YARN-408 URL: https://issues.apache.org/jira/browse/YARN-408 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Mayank Bansal Assignee: Mayank Bansal Priority: Minor Attachments: YARN-408-trunk.patch Capacity Scheduler delay scheduling should not be disabled by default. Enabling it to number of nodes in one rack. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-409) Allow apps to be killed via the RM REST API
Sandy Ryza created YARN-409: --- Summary: Allow apps to be killed via the RM REST API Key: YARN-409 URL: https://issues.apache.org/jira/browse/YARN-409 Project: Hadoop YARN Issue Type: New Feature Components: api, resourcemanager Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza The RM REST API currently allows getting information about running applications. Adding the capability to kill applications would allow systems like Hue to perform their functions over HTTP. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
