Thanks, so I'll check YARN. Does anyone know if Spark-on-Yarn plans to expose such functionality?
сб, 19 янв. 2019 г. в 18:04, Felix Cheung <felixcheun...@hotmail.com>: > To clarify, yarn actually supports excluding node right when requesting > resources. It’s spark that doesn’t provide a way to populate such a > blacklist. > > If you can change yarn config, the equivalent is node label: > https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/NodeLabel.html > > > > ------------------------------ > *From:* Li Gao <ligao...@gmail.com> > *Sent:* Saturday, January 19, 2019 8:43 AM > *To:* Felix Cheung > *Cc:* Serega Sheypak; user > *Subject:* Re: Spark on Yarn, is it possible to manually blacklist nodes > before running spark job? > > on yarn it is impossible afaik. on kubernetes you can use taints to keep > certain nodes outside of spark > > On Fri, Jan 18, 2019 at 9:35 PM Felix Cheung <felixcheun...@hotmail.com> > wrote: > >> Not as far as I recall... >> >> >> ------------------------------ >> *From:* Serega Sheypak <serega.shey...@gmail.com> >> *Sent:* Friday, January 18, 2019 3:21 PM >> *To:* user >> *Subject:* Spark on Yarn, is it possible to manually blacklist nodes >> before running spark job? >> >> Hi, is there any possibility to tell Scheduler to blacklist specific >> nodes in advance? >> >
