[jira] [Commented] (YARN-424) Allow resource requests to provide hints to the scheduler
[ https://issues.apache.org/jira/browse/YARN-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585965#comment-13585965 ] Alejandro Abdelnur commented on YARN-424: - Regarding the format/type for hints, while using 16 bytes would do, I'd prefer to use something that does not require having to deal with byte masking. While chatting about this, Bikas suggested a byte[], that woud be certainly much better. Still, I think having a capped String (50 or 100 chars) would something much easier to deal with the plus that printing a hint in the logs would be straight forward. My take is that hints should be a comma separated list of key/values or values * HINTS - ΓΈ | HINT | HINTS,HINT * HINT - VALUE | KEY=VALUE * KEY - word * VALUE - word Allow resource requests to provide hints to the scheduler -- Key: YARN-424 URL: https://issues.apache.org/jira/browse/YARN-424 Project: Hadoop YARN Issue Type: Improvement Components: api, applications, scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza As initially discussed on YARN-392, it would be helpful for the allocate protocol to be able to include hints to the scheduler. Examples hints could include deadlines (containers needed by a certain time) or gangs (resources that should be scheduled at the same time or not at all). To avoid placing complex scheduling expectations on the RM, the restrictions on hints would be: * Any scheduler may ignore any subset of hints. That is, applications must be written to handle resource requests as if their hints don't exist. * Must be bounded in length - perhaps no larger than 16 bytes or 100 characters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-424) Allow resource requests to provide hints to the scheduler
[ https://issues.apache.org/jira/browse/YARN-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586526#comment-13586526 ] Arun C Murthy commented on YARN-424: I'm not a fan of stringly-typed 'features' in the Scheduler. We should focus on actual use-cases and come up with well thought out solutions to those - MapReduce configuration is a very good example of how using key/value pairs leads us down to crazy nooks and corners. Gang-scheduling, deadline-scheduling etc. are core-features in the scheduler, and should be treated as such i.e. we should explicitly support them in the protocol - not as string hints. Allow resource requests to provide hints to the scheduler -- Key: YARN-424 URL: https://issues.apache.org/jira/browse/YARN-424 Project: Hadoop YARN Issue Type: Improvement Components: api, applications, scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza As initially discussed on YARN-392, it would be helpful for the allocate protocol to be able to include hints to the scheduler. Examples hints could include deadlines (containers needed by a certain time) or gangs (resources that should be scheduled at the same time or not at all). To avoid placing complex scheduling expectations on the RM, the restrictions on hints would be: * Any scheduler may ignore any subset of hints. That is, applications must be written to handle resource requests as if their hints don't exist. * Must be bounded in length - perhaps no larger than 16 bytes or 100 characters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-424) Allow resource requests to provide hints to the scheduler
[ https://issues.apache.org/jira/browse/YARN-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586535#comment-13586535 ] Arun C Murthy commented on YARN-424: Also, the reason we have invested in using protobufs (or any sane serialization system) is that we can *add* features to the Scheduler protocol in a compatible manner when we need to - key/value pairs are evil, and completely unnecessary. I've had enough with key/value pairs, and then attendant demands to maintain compatibility for those key/value pairs forever. Allow resource requests to provide hints to the scheduler -- Key: YARN-424 URL: https://issues.apache.org/jira/browse/YARN-424 Project: Hadoop YARN Issue Type: Improvement Components: api, applications, scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza As initially discussed on YARN-392, it would be helpful for the allocate protocol to be able to include hints to the scheduler. Examples hints could include deadlines (containers needed by a certain time) or gangs (resources that should be scheduled at the same time or not at all). To avoid placing complex scheduling expectations on the RM, the restrictions on hints would be: * Any scheduler may ignore any subset of hints. That is, applications must be written to handle resource requests as if their hints don't exist. * Must be bounded in length - perhaps no larger than 16 bytes or 100 characters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-424) Allow resource requests to provide hints to the scheduler
[ https://issues.apache.org/jira/browse/YARN-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586560#comment-13586560 ] Bikas Saha commented on YARN-424: - I agree that being able to extend the API without breaking back-compat is important. I am +1 for Arun's argument about protobuf's being the right approach to that. There is no need to add a string or any other hint. Simple add an optional member to the AllocateRequest which the new scheduler will understand. Other schedulers simply dont know about the new field. Allow resource requests to provide hints to the scheduler -- Key: YARN-424 URL: https://issues.apache.org/jira/browse/YARN-424 Project: Hadoop YARN Issue Type: Improvement Components: api, applications, scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza As initially discussed on YARN-392, it would be helpful for the allocate protocol to be able to include hints to the scheduler. Examples hints could include deadlines (containers needed by a certain time) or gangs (resources that should be scheduled at the same time or not at all). To avoid placing complex scheduling expectations on the RM, the restrictions on hints would be: * Any scheduler may ignore any subset of hints. That is, applications must be written to handle resource requests as if their hints don't exist. * Must be bounded in length - perhaps no larger than 16 bytes or 100 characters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
