Steve Loughran commented on YARN-3337:

the slider chaos monkey is pretty sophisticated as it is configured/deployed 
from within the AM itself, can trigger container failure and AM failure itself. 

I'm proposing something more minimal here, a CLI tool/class that can look up an 
app by ID or (user, type, name) and repeatedly sleep, decide whether or not to 
act, and act. Initial actions: kill the AM container, kill other containers.

I don't think the current client API lets me do this, as the tool would need
# ability to kill a specific container of an application, ideally forcing exit 
# ability to identify which container the AM is currently running in (so as not 
to kill it in a "worker container kill" operation, but do kill it in an AM kill 

In SLIDER-202 we are doing this in-AM, which has the data & operations; this is 
a visible change to the code which could easily be handled in a re-usable 
lib/CLI tool for better YARN app testing

> Provide YARN chaos monkey
> -------------------------
>                 Key: YARN-3337
>                 URL: https://issues.apache.org/jira/browse/YARN-3337
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 2.7.0
>            Reporter: Steve Loughran
> To test failure resilience today you either need custom scripts or implement 
> Chaos Monkey-like logic in your application (SLIDER-202). 
> Killing AMs and containers on a schedule & probability is the core activity 
> here, one that could be handled by a CLI App/client lib that does this. 
> # entry point to have a startup delay before acting
> # frequency of chaos wakeup/polling
> # probability to AM failure generation (0-100)
> # probability of non-AM container kill
> # future: other operations

This message was sent by Atlassian JIRA

Reply via email to