[ 
https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374757#comment-16374757
 ] 

Yinan Li edited comment on SPARK-23485 at 2/23/18 6:22 PM:
-----------------------------------------------------------

It's not that I'm too confident on the capability of Kubernetes to detect node 
problems. I just don't see it as a good practice of worrying about node 
problems at application level in a containerized environment running on a 
container orchestration system. For that reason, yes, I don't think Spark on 
Kubernetes should really need to worry about blacklisting nodes.


was (Author: liyinan926):
It's not that I'm too confident on the capability of Kubernetes to detect node 
problems. I just don't see it as a good practice of worrying about node 
problems at application level in a containerized environment running on a 
container orchestration system. Yes, I don't think Spark on Kubernetes should 
really need to worry about blacklisting nodes.

> Kubernetes should support node blacklist
> ----------------------------------------
>
>                 Key: SPARK-23485
>                 URL: https://issues.apache.org/jira/browse/SPARK-23485
>             Project: Spark
>          Issue Type: New Feature
>          Components: Kubernetes, Scheduler
>    Affects Versions: 2.3.0
>            Reporter: Imran Rashid
>            Priority: Major
>
> Spark's BlacklistTracker maintains a list of "bad nodes" which it will not 
> use for running tasks (eg., because of bad hardware).  When running in yarn, 
> this blacklist is used to avoid ever allocating resources on blacklisted 
> nodes: 
> https://github.com/apache/spark/blob/e836c27ce011ca9aef822bef6320b4a7059ec343/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala#L128
> I'm just beginning to poke around the kubernetes code, so apologies if this 
> is incorrect -- but I didn't see any references to 
> {{scheduler.nodeBlacklist()}} in {{KubernetesClusterSchedulerBackend}} so it 
> seems this is missing.  Thought of this while looking at SPARK-19755, a 
> similar issue on mesos.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to