On 8 January 2015 at 18:15, Nitin Aggarwal <[email protected]> wrote:
> +1. Also, should the node with failure threshold reached, be ever > considered in future for allocation ? > It is possible that due to some temporary issues with few nodes, locality > node set, could have a lot more nodes than needed. > There's various options here, whose policy depends on cluster size. a key one that's surfaced in MapReduce a long, long time ago: *if you only have one node, don't blacklist it.* in fact, for any small cluster you don't want to think about blacklisting so much as sorting them by reliability, and preferring those you consider more reliable. But, that's complicated. State of the art here is probably The φ Accrual Failure Detector: http://ddg.jaist.ac.jp/pub/HDY+04.pdf, I think what we can do at it simplest is just reset the counter every day. Even when we don't trust it enough to explicitly ask for containers on it, we could just say "anywhere" and have it picked up sometimes, especially on smaller clusters where there are less servers to choose from. There is also a blacklist API in YARN, which we've not looked at yet. It lets us tell YARN which nodes to blacklist, which to whitelist. We'd have to track which ones were blacklisted and whitelist them after a while, unless we want to end up blacklisting the entire cluster. so again: some kind of reset operation, though if we track when a node was blacklisted, we could give it a specific time interval (say 24h) to get better. All this failure tracking history is lost on an AM failure incidentally, possibly including the list of blacklisted nodes handed off to YARN. > > On Thu, Jan 8, 2015 at 9:19 AM, Steve Loughran <[email protected]> > wrote: > > > https://issues.apache.org/jira/browse/SLIDER-743 it is then. > > > > On 8 January 2015 at 14:26, Jon Maron <[email protected]> wrote: > > > > > +1. A good way to provide the functionality while leveraging existing > > > mechanisms > > > > > > On Jan 8, 2015, at 8:46 AM, Gour Saha <[email protected]> wrote: > > > > > > > +1 on that > > > > > > > > That's also what I meant when I said - > > > >>> I don't think we have a logic where we apply data locality and then > > > upon a > > > >>> certain no of failures (threshold) try with "no data locality" at > > least > > > >>> once before giving up. It will be a good idea to file a JIRA with > > this > > > >>> requirement. > > > > > > > > -Gour > > > > > > > > - Sent from my iPhone > > > > > > > >> On Jan 8, 2015, at 3:30 AM, Steve Loughran <[email protected]> > > > wrote: > > > >> > > > >> thinking about this some more, we could use our tracking of node > > > >> reliability to tune our placement decisions. > > > >> > > > >> > > > >> 1. We add a "recent failures" field to the node entries, alongside > > the > > > >> "total failures" > > > >> 2. Our scheduled failure count resetter will set that field to > zero, > > > >> alongside the component failures > > > >> 3. When Slider has to request a new container, unless the placement > > > >> policy is STRICT, we will continue to use the (persisted) placement > > > history > > > >> 4. Except now, if a node has a recent failure count above some > > > >> threshold, we don't ask for a container on that node...we just ask > > for > > > >> "anywhere" placement. > > > >> > > > >> What do people think? > > > >> > > > >>> On 7 January 2015 at 09:50, Steve Loughran <[email protected] > > > > > wrote: > > > >>> > > > >>> the history of where things were is retained in the RoleHistory > > > >>> structures, persisted to HDFS and reread on startup. for each > > component > > > >>> type, it's sorted by most-recent-first. > > > >>> > > > >>> When a container is needed, the AM looks in that history first, and > > > looks > > > >>> through the list of "previously used nodes for that component > type"., > > > >>> skipping any that already have an instance of that component > running. > > > The > > > >>> chosen node is taken off the list, so there's no duplicates > > > >>> (exception: the component type doesn't have any locality, in which > > case > > > >>> although the history is tracked, it's not used for placement) > > > >>> > > > >>> > > > >>> > > > >>> When a placement on the node comes in, then its taken off the > > "pending > > > >>> list" > > > >>> > > > >>> There's one small issue here: no way to tie requests to > allocations. > > We > > > >>> don't really care which request allocates a component to a node, we > > > just > > > >>> like to track outstanding requests for explicit nodes. The > algorithm > > is > > > >>> -allocation to a requested node: remove node from "list of > > outstanding > > > >>> explicit requests" > > > >>> -allocation to another node: do nothing while there are outstanding > > > >>> requests > > > >>> -all outstanding requests satisfied: clean the list of outstanding > > > >>> "placed" requests. > > > >>> > > > >>> Now, fun happens when a container fails on a newly allocated node > > —and > > > its > > > >>> here there may be some policy tuning required. > > > >>> > > > >>> It comes down to this: what is the best way to react when a > component > > > >>> fails to start, either immediately, or shortly after startup? This > > can > > > be a > > > >>> sign of a major problem "node doesn't run my app", or something > > > transient > > > >>> "port still considered in use" > > > >>> > > > >>> If its a transient problem, there's no harm in asking again. > > > >>> > > > >>> If its a permanent problem: we need to make the decision that this > > > node is > > > >>> bad —at least for that specific component. > > > >>> > > > >>> I think right now, on a startup/launch time failure, the failing > node > > > is > > > >>> placed at the back of the list of recently used nodes; the failure > > > counts > > > >>> of both the node and the component incremented. Although there's a > > > YARN API > > > >>> where an application can provide blacklist hints to YARN, we're not > > > >>> currently using it. > > > >>> > > > >>> I think what you may be seeing is that Slider is repeatedly asking > > for > > > the > > > >>> same node: it's failing and going to the back of the list of > > previously > > > >>> used nodes, but at there is only one, it's being asked for again. > > > >>> > > > >>> We can tune this -maybe- but it gets complex. > > > >>> > > > >>> 1. If the placement policy is STRICT, then we must ask for that > > > previously > > > >>> used node. (Though thinking about it, the component must have > started > > > at > > > >>> least once at some point in the past...I don't know if the special > > > case of > > > >>> "previously allocated but never started" is detected and handled) > > > >>> > > > >>> 2. If the placement is location-preferred, default, how best to > react > > > to a > > > >>> launch failure? Completely cut that node off the list of suitable > > > targets? > > > >>> Or try again a few more times? If its a transient problem, retry > > gives > > > >>> locality without over-reacting. If its a permanent problem, then > > > retrying > > > >>> is the wrong policy. > > > >>> > > > >>> What should we do here? We are tracking failures in NodeEntry > > entries, > > > in > > > >>> a map of the cluster built up (NodeMap), but not currently using > > > failure > > > >>> counts there to make decisions. If we do think about using it, > we'll > > > have > > > >>> to think about not just keeping the count of failures, but > resetting > > > it on > > > >>> an interval, the way we now do with component failure counts. > > > >>> > > > >>> -steve > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>>> On 7 January 2015 at 02:50, Gour Saha <[email protected]> > > wrote: > > > >>>> > > > >>>> Nitin, > > > >>>> > > > >>>> I don't think we have a logic where we apply data locality and > then > > > upon a > > > >>>> certain no of failures (threshold) try with "no data locality" at > > > least > > > >>>> once before giving up. It will be a good idea to file a JIRA with > > this > > > >>>> requirement. > > > >>>> > > > >>>> -Gour > > > >>>> > > > >>>> > > > >>>> On Tue, Jan 6, 2015 at 5:12 PM, Nitin Aggarwal < > > > >>>> [email protected] > > > >>>>> wrote: > > > >>>> > > > >>>>> I am running HBase application, and I prefer data locality. I > don't > > > >>>> want to > > > >>>>> give up locality by default. It's ok to lose locality in rare > > > scenarios, > > > >>>>> where something is wrong with one of the local nodes. > > > >>>>> It's more of fail-safe that I am looking for, to give up > locality, > > > if it > > > >>>>> cannot be satisfied. > > > >>>>> > > > >>>>> Thanks > > > >>>>> Nitin > > > >>>>> > > > >>>>> > > > >>>>>> On Tue, Jan 6, 2015 at 4:52 PM, Ted Yu <[email protected]> > > wrote: > > > >>>>>> > > > >>>>>> Here is the meaning of 2 (see PlacementPolicy): > > > >>>>>> > > > >>>>>> * No data locality; do not bother trying to ask for any > location > > > >>>>>> > > > >>>>>> */ > > > >>>>>> > > > >>>>>> public static final int NO_DATA_LOCALITY = 2; > > > >>>>>> > > > >>>>>> On Tue, Jan 6, 2015 at 4:15 PM, Gour Saha < > [email protected]> > > > >>>> wrote: > > > >>>>>> > > > >>>>>>> Try setting property *yarn.component.placement.policy* to 2 for > > the > > > >>>>>>> component, something like this - > > > >>>>>>> > > > >>>>>>> "HBASE_MASTER": { > > > >>>>>>> "yarn.role.priority": "1", > > > >>>>>>> "yarn.component.instances": "1", > > > >>>>>>> "yarn.memory": "1500", > > > >>>>>>> "yarn.component.placement.policy": "2" > > > >>>>>>> }, > > > >>>>>>> > > > >>>>>>> -Gour > > > >>>>>>> > > > >>>>>>> On Tue, Jan 6, 2015 at 3:33 PM, Nitin Aggarwal < > > > >>>>>>> [email protected] > > > >>>>>>>> wrote: > > > >>>>>>> > > > >>>>>>>> Hi, > > > >>>>>>>> > > > >>>>>>>> We keep on running into scenario, where one of the node in the > > > >>>>> cluster > > > >>>>>>> went > > > >>>>>>>> bad (either due to clock out of sync, no disk space etc.). As > a > > > >>>>> result > > > >>>>>>>> container fails to start, and due to locality, container is > > > >>>> assigned > > > >>>>> on > > > >>>>>>> the > > > >>>>>>>> same machine again and again, and it fails again and again. > > After > > > >>>> few > > > >>>>>>>> failures, when failure threshold is reached (which is > currently > > > >>>> also > > > >>>>>> not > > > >>>>>>>> reset correctly. SLIDER-629), it triggers instance shut-down. > > > >>>>>>>> > > > >>>>>>>> Is there a way to give up locality, in case of multiple > > failures, > > > >>>> to > > > >>>>>>> avoid > > > >>>>>>>> this scenario ? > > > >>>>>>>> > > > >>>>>>>> Thanks > > > >>>>>>>> Nitin Aggarwal > > > >>>>>>> > > > >>>>>>> -- > > > >>>>>>> CONFIDENTIALITY NOTICE > > > >>>>>>> NOTICE: This message is intended for the use of the individual > or > > > >>>>> entity > > > >>>>>> to > > > >>>>>>> which it is addressed and may contain information that is > > > >>>> confidential, > > > >>>>>>> privileged and exempt from disclosure under applicable law. If > > the > > > >>>>> reader > > > >>>>>>> of this message is not the intended recipient, you are hereby > > > >>>> notified > > > >>>>>> that > > > >>>>>>> any printing, copying, dissemination, distribution, disclosure > or > > > >>>>>>> forwarding of this communication is strictly prohibited. If you > > > have > > > >>>>>>> received this communication in error, please contact the sender > > > >>>>>> immediately > > > >>>>>>> and delete it from your system. Thank You. > > > >>>> > > > >>>> -- > > > >>>> CONFIDENTIALITY NOTICE > > > >>>> NOTICE: This message is intended for the use of the individual or > > > entity > > > >>>> to > > > >>>> which it is addressed and may contain information that is > > > confidential, > > > >>>> privileged and exempt from disclosure under applicable law. If the > > > reader > > > >>>> of this message is not the intended recipient, you are hereby > > notified > > > >>>> that > > > >>>> any printing, copying, dissemination, distribution, disclosure or > > > >>>> forwarding of this communication is strictly prohibited. If you > have > > > >>>> received this communication in error, please contact the sender > > > >>>> immediately > > > >>>> and delete it from your system. Thank You. > > > >> > > > >> -- > > > >> CONFIDENTIALITY NOTICE > > > >> NOTICE: This message is intended for the use of the individual or > > > entity to > > > >> which it is addressed and may contain information that is > > confidential, > > > >> privileged and exempt from disclosure under applicable law. If the > > > reader > > > >> of this message is not the intended recipient, you are hereby > notified > > > that > > > >> any printing, copying, dissemination, distribution, disclosure or > > > >> forwarding of this communication is strictly prohibited. If you have > > > >> received this communication in error, please contact the sender > > > immediately > > > >> and delete it from your system. Thank You. > > > > > > > > -- > > > > CONFIDENTIALITY NOTICE > > > > NOTICE: This message is intended for the use of the individual or > > entity > > > to > > > > which it is addressed and may contain information that is > confidential, > > > > privileged and exempt from disclosure under applicable law. If the > > reader > > > > of this message is not the intended recipient, you are hereby > notified > > > that > > > > any printing, copying, dissemination, distribution, disclosure or > > > > forwarding of this communication is strictly prohibited. If you have > > > > received this communication in error, please contact the sender > > > immediately > > > > and delete it from your system. Thank You. > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
