I think “exclude nodes” and such is really the job of the resource manager i.e.
Yarn. So I am not sure taking over some of these tasks in Apex would be very
useful.
I agree with Amol that apps should be node neutral. Resource management in Yarn
together with fault tolerance in Apex should minimize the need for this feature
although I am sure one can find use cases.
On 11/29/16, 10:41 PM, "Amol Kekre" <a...@datatorrent.com> wrote:
We do have this feature in Yarn, but that applies to all applications. I am
not sure if Yarn has anti-affinity. This feature may be used, but in
general there is danger is an application taking over resource allocation.
Another quirk is that big data apps should ideally be node-neutral. This is
a good idea, if we are able to carve out something where need is app
specific.
Thks
Amol
On Tue, Nov 29, 2016 at 10:00 PM, Milind Barve <mili...@gmail.com> wrote:
> We have seen 2 cases mentioned below, where, it would have been nice if
> Apex allowed us to exclude a node from the cluster for an application.
>
> 1. A node in the cluster had gone bad (was randomly rebooting) and so an
> Apex app should not use it - other apps can use it as they were batch
jobs.
> 2. A node is being used for a mission critical app (Could be an Apex app
> itself), but another Apex app which is mission critical should not be
using
> resources on that node.
>
> Can we have a way in which, Stram and YARN can coordinate between each
> other to not use a set of nodes for the application. It an be done in 2
way
> s-
>
> 1. Have a list of "exclude" nodes with Stram- when YARN allcates resources
> on either of these, STRAM rejects and gets resources allocated again frm
> YARN
> 2. Have a list of nodes that can be used for an app - This can be a part
of
> config. Hwever, I don't think this would be a right way to do so as we
will
> need support from YARN as well. Further, this might be difficult to change
> at runtim if need be.
>
> Any thoughts?
>
>
> --
> ~Milind bee at gee mail dot com
>
