[jira] [Commented] (YARN-664) throw InvalidRequestException for requests with different capabilities at the same priority
[ https://issues.apache.org/jira/browse/YARN-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664760#comment-13664760 ] Arun C Murthy commented on YARN-664: +1 for fixing this. This is, potentially, a very bad bug. I'm not convinced about YARN-314, so for now, at least, we should throw an exception rather than cause inconsistencies. throw InvalidRequestException for requests with different capabilities at the same priority --- Key: YARN-664 URL: https://issues.apache.org/jira/browse/YARN-664 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Nothing stops an application from submitting a request with priority=1, location=*, memory=1024 and a request with priority=1, location=rack1, memory=2048. However, this does not make sense under the request model and can cause bad things to happen in the scheduler. It should be possible to detect this at AMRM heartbeat time and throw an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-664) throw InvalidRequestException for requests with different capabilities at the same priority
[ https://issues.apache.org/jira/browse/YARN-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664808#comment-13664808 ] Bikas Saha commented on YARN-664: - I don't think that this is invalid as far as the API is concerned. There might be reasons in the current implementations of the schedulers that may make supporting this prohibitive, and so it was not done. Say an application is running 2 concurrent but unrelated items that should not be blocked by each other due to performance. Both have different resources sizes. If they are separated by priority then the lower priority item will not get any containers until all the containers for the higher priority item have been allocated. This is by definition of priority. That may lead to poor performance. So the app would want to run both at the same priority and expect the RM to allocate containers to both wherever resources are free to match them. The most simple solution for the RM that I can think off the top of my head is to always use the largest resource size. It may lead to fragmentation but is safe. Currently, the last resource size is used. This means that if the client asks for 1GB and 2GB and the 1GB happens to be the last one set by the RM then the RM will return 2 1GB containers, which is wrong. If we choose to use the largest value as a solution, then we should close this as invalid and use YARN-314 to implement that solution. throw InvalidRequestException for requests with different capabilities at the same priority --- Key: YARN-664 URL: https://issues.apache.org/jira/browse/YARN-664 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Nothing stops an application from submitting a request with priority=1, location=*, memory=1024 and a request with priority=1, location=rack1, memory=2048. However, this does not make sense under the request model and can cause bad things to happen in the scheduler. It should be possible to detect this at AMRM heartbeat time and throw an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-664) throw InvalidRequestException for requests with different capabilities at the same priority
[ https://issues.apache.org/jira/browse/YARN-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657452#comment-13657452 ] Sandy Ryza commented on YARN-664: - My mistake - the original example was valid, just switched it to be an invalid one. The AMRM protocol expects a corresponding rack-level and *-level request for every node-level request, and a corresponding *-level request for every rack-level request. The requests in the updated example now mean something like if you give me a container on rack1, it should have 2048 MB, but if it's on any other rack, it should have 1024 MB. I don't think the scheduler should support this, as it's hard to imagine when it would be necessary, and makes it impossible to calculate an application's demand. If what you're saying is that the scheduler should be able to support requests like: priority=1, location=*, memory=1024 priority=1, location=rack1, memory=1024 priority=1, location=*, memory=2048 priority=1, location=rack1, memory=2048 that might make sense. I had filed YARN-314 for this a while ago, but have since become less convinced of its utility. It would require deepening the data structures in the scheduler, which would mean extra hash lookups for each request. The philosophy I've perceived in the design has been that containers with different requirements should be requested at explicitly different priorities. [~acm], would you be able to weigh in? throw InvalidRequestException for requests with different capabilities at the same priority --- Key: YARN-664 URL: https://issues.apache.org/jira/browse/YARN-664 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Nothing stops an application from submitting a request with priority=1, location=*, memory=1024 and a request with priority=1, location=rack1, memory=2048. However, this does not make sense under the request model and can cause bad things to happen in the scheduler. It should be possible to detect this at AMRM heartbeat time and throw an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-664) throw InvalidRequestException for requests with different capabilities at the same priority
[ https://issues.apache.org/jira/browse/YARN-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657453#comment-13657453 ] Sandy Ryza commented on YARN-664: - Sorry, [~acm], meant to mention [~acmurthy], not you. throw InvalidRequestException for requests with different capabilities at the same priority --- Key: YARN-664 URL: https://issues.apache.org/jira/browse/YARN-664 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Nothing stops an application from submitting a request with priority=1, location=*, memory=1024 and a request with priority=1, location=rack1, memory=2048. However, this does not make sense under the request model and can cause bad things to happen in the scheduler. It should be possible to detect this at AMRM heartbeat time and throw an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-664) throw InvalidRequestException for requests with different capabilities at the same priority
[ https://issues.apache.org/jira/browse/YARN-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13654981#comment-13654981 ] Hitesh Shah commented on YARN-664: -- Actually, isn't the example actually a valid request? throw InvalidRequestException for requests with different capabilities at the same priority --- Key: YARN-664 URL: https://issues.apache.org/jira/browse/YARN-664 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Nothing stops an application from submitting a request with priority=1, location=*, memory=1024 and a request with priority=1, location=rack1, memory=1024. However, this does not make sense under the request model and can cause bad things to happen in the scheduler. It should be possible to detect this at AMRM heartbeat time and throw an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-664) throw InvalidRequestException for requests with different capabilities at the same priority
[ https://issues.apache.org/jira/browse/YARN-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13654986#comment-13654986 ] Hitesh Shah commented on YARN-664: -- Original comment still holds for an AM being able to request 1 container of 1 GB and 1 container of 4 GB both at priority 1. throw InvalidRequestException for requests with different capabilities at the same priority --- Key: YARN-664 URL: https://issues.apache.org/jira/browse/YARN-664 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Nothing stops an application from submitting a request with priority=1, location=*, memory=1024 and a request with priority=1, location=rack1, memory=1024. However, this does not make sense under the request model and can cause bad things to happen in the scheduler. It should be possible to detect this at AMRM heartbeat time and throw an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
