[ 
https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373652#comment-16373652
 ] 

Stavros Kontopoulos edited comment on SPARK-23485 at 2/23/18 12:04 AM:
-----------------------------------------------------------------------

Does the scheduler know all the reasons an app might want to define a node as 
problematic and (blacklisted in our case)? Is there a pointer for the node 
problems ?

It seems to me that this is the reason taints are introduced (might be wrong). 

Could kubelet detect the scenario for which blacklist was introduced anyway:

[https://blog.cloudera.com/blog/2017/04/blacklisting-in-apache-spark/?|https://blog.cloudera.com/blog/2017/04/blacklisting-in-apache-spark/]
 


was (Author: skonto):
Does the scheduler know all the reasons an app might want to define a node as 
problematic and (blacklisted in our case)? Is there a pointer for the node 
problems ?

It seems to me that this is the reason taints are introduced (might be wrong). 

> Kubernetes should support node blacklist
> ----------------------------------------
>
>                 Key: SPARK-23485
>                 URL: https://issues.apache.org/jira/browse/SPARK-23485
>             Project: Spark
>          Issue Type: New Feature
>          Components: Kubernetes, Scheduler
>    Affects Versions: 2.3.0
>            Reporter: Imran Rashid
>            Priority: Major
>
> Spark's BlacklistTracker maintains a list of "bad nodes" which it will not 
> use for running tasks (eg., because of bad hardware).  When running in yarn, 
> this blacklist is used to avoid ever allocating resources on blacklisted 
> nodes: 
> https://github.com/apache/spark/blob/e836c27ce011ca9aef822bef6320b4a7059ec343/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala#L128
> I'm just beginning to poke around the kubernetes code, so apologies if this 
> is incorrect -- but I didn't see any references to 
> {{scheduler.nodeBlacklist()}} in {{KubernetesClusterSchedulerBackend}} so it 
> seems this is missing.  Thought of this while looking at SPARK-19755, a 
> similar issue on mesos.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to