[jira] [Commented] (YARN-6486) FairScheduler: Deprecate continuous scheduling in 2.9
[ https://issues.apache.org/jira/browse/YARN-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009615#comment-16009615 ] Carlo Curino commented on YARN-6486: I agree that if this implementation is simply pretending to see many HBs to trigger the rest of the logic, there are more scalable ways to go about it. [~asuresh] it seems that the kind of ideas you have been working on are being referred to as "Global scheduling" in this context (i.e., non-HB based decision making). Given some prod experience we have, it seems that time-based locality also helps opening up alternative optimization strategies, so I am all for that. As it stands I don't have anything against deprecating this, [~asuresh] please look at the code, and see whether you think some of it should be saved for the purposes you have in mind. > FairScheduler: Deprecate continuous scheduling in 2.9 > - > > Key: YARN-6486 > URL: https://issues.apache.org/jira/browse/YARN-6486 > Project: Hadoop YARN > Issue Type: Task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg > > Mark continuous scheduling as deprecated in 2.9 and remove the code in 3.0. > Removing continuous scheduling from the code will be logged as a separate jira -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6486) FairScheduler: Deprecate continuous scheduling in 2.9
[ https://issues.apache.org/jira/browse/YARN-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005704#comment-16005704 ] Karthik Kambatla commented on YARN-6486: The non-HB driven approach allows certain optimizations. However, in our experience, the current implementation doesn't scale well. Current implementation continuously simulates node heartbeats and exposes lock contention issues in the scheduler. We have been recommending turning this off for large clusters. Global scheduling is likely the better approach that is not driven by heartbeats. I am open to less-lock-contention approaches. > FairScheduler: Deprecate continuous scheduling in 2.9 > - > > Key: YARN-6486 > URL: https://issues.apache.org/jira/browse/YARN-6486 > Project: Hadoop YARN > Issue Type: Task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg > > Mark continuous scheduling as deprecated in 2.9 and remove the code in 3.0. > Removing continuous scheduling from the code will be logged as a separate jira -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6486) FairScheduler: Deprecate continuous scheduling in 2.9
[ https://issues.apache.org/jira/browse/YARN-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005078#comment-16005078 ] Arun Suresh commented on YARN-6486: --- Any reason why you would want to deprecate this ? Think we should unify Async Scheduling (from the Capacity Scheduler) and Continuous scheduling, since it looks pretty similar. Atleast the triggering thread can be made a common RM service. There are some interesting scheduling performance improvements that can be done using a non-HB driven approach, coupled with time-based localilty relaxation that I've been experimenting with, both of which are offered by the Fair Scheduler. it would be sad to see this deprecated as a feature. Thoughts ? (cc: [~ka...@cloudera.com], [~subru], [~leftnoteasy], [~curino]) > FairScheduler: Deprecate continuous scheduling in 2.9 > - > > Key: YARN-6486 > URL: https://issues.apache.org/jira/browse/YARN-6486 > Project: Hadoop YARN > Issue Type: Task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg > > Mark continuous scheduling as deprecated in 2.9 and remove the code in 3.0. > Removing continuous scheduling from the code will be logged as a separate jira -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
