Yeah, trying to do the injection client-side to avoid disruption to other users (and having to deploy/reconfigure HDFS). I was hoping someone has already created that :) We will probably create it at some point and may try to submit a patch later.
-----Original Message----- From: Stephen Loughran <ste...@cloudera.com.INVALID> Sent: Tuesday, February 12, 2019 3:33 PM To: Sergey Shelukhin <sergey.sheluk...@microsoft.com.invalid> Cc: hdfs-dev@hadoop.apache.org Subject: Re: DFSClient/DistriburedFileSystem fault injection? Sergey -you trying to simulate failures client side, or do you have an NN Which actually injects failures all the way up the IPC stack? as if its just client, couldn't registering a fault-injecting client as fs.hdfs.impl could do that FWIW, in the s3a connector we have the "inconsistent" s3 client which mimics some symptoms of delayed consistency; it has a path, a probability of happening and a delay before things become visible. This is in the main hadoop-aws JAR, and is turned on by a configuration switch (yes, it prints a big warning). With a single switch to turn it on, its trivial to enable it in tests On Mon, Feb 11, 2019 at 11:42 PM Sergey Shelukhin <sergey.sheluk...@microsoft.com.invalid> wrote: > Hi. > I've been looking for a client-side solution for fault injection in HDFS. > We had a naturally unstable HDFS cluster that helped uncover a lot of > issues in HBase; now that it has been stabilized, we miss it already > :) > > To keep testing without actually disrupting others' use of HDFS or > having to deploy a new version, I was thinking about having a > client-side schema (e.g. fhdfs) map to a wrapper over the standard DFS > that would inject failures and delays according to some configs, > similar to > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhado > op.apache.org%2Fdocs%2Fr2.7.2%2Fhadoop-project-dist%2Fhadoop-hdfs%2FFa > ultInjectFramework.html&data=02%7C01%7CSergey.Shelukhin%40microsof > t.com%7Cd586bd648a164a9dd45108d69142773e%7C72f988bf86f141af91ab2d7cd01 > 1db47%7C1%7C1%7C636856111994781945&sdata=9hiTYoOeDiKnz%2FNAvSRl%2F > AhqXtdIF%2FQcUwzQjorfJNU%3D&reserved=0 > > However I wonder if something like this exists already? > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
