Hi Imran, here is my usecase There is 1K nodes cluster and jobs have performance degradation because of a single node. It's rather hard to convince Cluster Ops to decommission node because of "performance degradation". Imagine 10 dev teams chase single ops team for valid reason (node has problems) or because code has a bug or data is skewed or spots on the sun. We can't just decommission node because random dev complains.
Simple solution: - rerun failed / delayed job and blacklist "problematic" node in advance. - Report about the problem if job works w/o anomalies. - ops collect complains about node and start to decommission it when "complains threshold" is reached. It's a rather low probability that many loosely coupled teams with loosely coupled jobs complain about a single node. Results - Ops are not spammed with a random requests from devs - devs are not blocked because of the really bad node. - it's very cheap for everyone to "blacklist" node during job submission w/o doing anything to node. - it's very easy to automate such behavior. Many teams use 100500 kinds of workflow runners and the strategy is dead simple (depends on SLA of course). - Just re-run failed job excluding nodes with failed tasks (if number of nodes is reasonable) - Kill stuck job if it runs longer than XXX minutes and re-start excluding nodes with long-running tasks. ср, 23 янв. 2019 г. в 23:09, Imran Rashid <iras...@cloudera.com>: > Serga, can you explain a bit more why you want this ability? > If the node is really bad, wouldn't you want to decomission the NM > entirely? > If you've got heterogenous resources, than nodelabels seem like they would > be more appropriate -- and I don't feel great about adding workarounds for > the node-label limitations into blacklisting. > > I don't want to be stuck supporting a configuration with too limited a use > case. > > (may be better to move discussion to > https://issues.apache.org/jira/browse/SPARK-26688 so its better archived, > I'm responding here in case you aren't watching that issue) > > On Tue, Jan 22, 2019 at 6:09 AM Jörn Franke <jornfra...@gmail.com> wrote: > >> You can try with Yarn node labels: >> >> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeLabel.html >> >> Then you can whitelist nodes. >> >> Am 19.01.2019 um 00:20 schrieb Serega Sheypak <serega.shey...@gmail.com>: >> >> Hi, is there any possibility to tell Scheduler to blacklist specific >> nodes in advance? >> >>
