[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752760#comment-15752760 ] Apache Spark commented on SPARK-8425: - User 'squito' has created a pull request for this issue: https://github.com/apache/spark/pull/16298 > Add blacklist mechanism for task scheduling > --- > > Key: SPARK-8425 > URL: https://issues.apache.org/jira/browse/SPARK-8425 > Project: Spark > Issue Type: Improvement > Components: Scheduler, YARN >Reporter: Saisai Shao >Assignee: Mao, Wei >Priority: Minor > Fix For: 2.2.0 > > Attachments: DesignDocforBlacklistMechanism.pdf > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524811#comment-15524811 ] Imran Rashid commented on SPARK-8425: - Breaking off a smaller chunk of this that can be added indepently in SPARK-17675 > Add blacklist mechanism for task scheduling > --- > > Key: SPARK-8425 > URL: https://issues.apache.org/jira/browse/SPARK-8425 > Project: Spark > Issue Type: Improvement > Components: Scheduler, YARN >Reporter: Saisai Shao >Assignee: Imran Rashid >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384719#comment-15384719 ] Imran Rashid commented on SPARK-8425: - [~tgraves] I don't know of any jira for that. Sounds like you already hit this case, so I'll let you open a jira as you might be able to provide more info, but please add me to it. > Add blacklist mechanism for task scheduling > --- > > Key: SPARK-8425 > URL: https://issues.apache.org/jira/browse/SPARK-8425 > Project: Spark > Issue Type: Improvement > Components: Scheduler, YARN >Reporter: Saisai Shao >Assignee: Imran Rashid >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379361#comment-15379361 ] Thomas Graves commented on SPARK-8425: -- Slightly different scenario since its not with the scheduling, but do we have any jira for blacklisting a node where containers/executors won't launch on it? > Add blacklist mechanism for task scheduling > --- > > Key: SPARK-8425 > URL: https://issues.apache.org/jira/browse/SPARK-8425 > Project: Spark > Issue Type: Improvement > Components: Scheduler, YARN >Reporter: Saisai Shao >Assignee: Imran Rashid >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365244#comment-15365244 ] Apache Spark commented on SPARK-8425: - User 'squito' has created a pull request for this issue: https://github.com/apache/spark/pull/14079 > Add blacklist mechanism for task scheduling > --- > > Key: SPARK-8425 > URL: https://issues.apache.org/jira/browse/SPARK-8425 > Project: Spark > Issue Type: Improvement > Components: Scheduler, YARN >Reporter: Saisai Shao >Assignee: Imran Rashid >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364834#comment-15364834 ] Thomas Graves commented on SPARK-8425: -- Added some questions to the design doc > Add blacklist mechanism for task scheduling > --- > > Key: SPARK-8425 > URL: https://issues.apache.org/jira/browse/SPARK-8425 > Project: Spark > Issue Type: Improvement > Components: Scheduler, YARN >Reporter: Saisai Shao >Assignee: Imran Rashid >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353587#comment-15353587 ] Apache Spark commented on SPARK-8425: - User 'squito' has created a pull request for this issue: https://github.com/apache/spark/pull/13951 > Add blacklist mechanism for task scheduling > --- > > Key: SPARK-8425 > URL: https://issues.apache.org/jira/browse/SPARK-8425 > Project: Spark > Issue Type: Improvement > Components: Scheduler, YARN >Reporter: Saisai Shao >Assignee: Imran Rashid >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353277#comment-15353277 ] Imran Rashid commented on SPARK-8425: - Adding the design doc from SPARK-8426 here: https://docs.google.com/document/d/1R2CVKctUZG9xwD67jkRdhBR4sCgccPR2dhTYSRXFEmg/edit?usp=sharing Also I want to point a change in behavior I am proposing in the design doc -- I think its best if there is *no timeout* for the blacklist *within one stage*. Once a task gets blacklisted for a particular stage, it will there forever. The timeout will only be for when executors and nodes get blacklisted across *all* stages. This greatly simplifies the implementation, and I dont' really think there is any significant downside. OTOH, it *is* a behavior change from the old blacklisting. > Add blacklist mechanism for task scheduling > --- > > Key: SPARK-8425 > URL: https://issues.apache.org/jira/browse/SPARK-8425 > Project: Spark > Issue Type: Improvement > Components: Scheduler, YARN >Reporter: Saisai Shao >Assignee: Imran Rashid >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14718204#comment-14718204 ] Mao, Wei commented on SPARK-8425: - Here is the new design doc based on Saisai's previous work with some enhancement. https://docs.google.com/document/d/1EqdocdbOH0eZ0Vp1RAHsE-8gKv9yPez1Xt8W5xgXn3I/edit?usp=sharing Add blacklist mechanism for task scheduling --- Key: SPARK-8425 URL: https://issues.apache.org/jira/browse/SPARK-8425 Project: Spark Issue Type: Improvement Components: Scheduler, YARN Reporter: Saisai Shao Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627365#comment-14627365 ] Saisai Shao commented on SPARK-8425: Repost the design doc here: https://docs.google.com/document/d/1oibmfcyewy_kBLjGwVbGAOXZGlWyWADJu1TbBc06sdk/edit?usp=sharing Add blacklist mechanism for task scheduling --- Key: SPARK-8425 URL: https://issues.apache.org/jira/browse/SPARK-8425 Project: Spark Issue Type: Improvement Components: Scheduler, YARN Reporter: Saisai Shao Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591305#comment-14591305 ] Apache Spark commented on SPARK-8425: - User 'jerryshao' has created a pull request for this issue: https://github.com/apache/spark/pull/6870 Add blacklist mechanism for task scheduling --- Key: SPARK-8425 URL: https://issues.apache.org/jira/browse/SPARK-8425 Project: Spark Issue Type: Sub-task Components: Scheduler, YARN Reporter: Saisai Shao -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
