+1 (binding) Great work guys! On Thu, Aug 24, 2017 at 5:01 AM, Steve Loughran <ste...@hortonworks.com> wrote:
> > On 23 Aug 2017, at 19:21, Aaron Fabbri <fab...@cloudera.com<mailto:fa > b...@cloudera.com>> wrote: > > > On Tue, Aug 22, 2017 at 10:24 AM, Steve Loughran <ste...@hortonworks.com< > mailto:ste...@hortonworks.com>> wrote: > video being processed: https://www.youtube.com/watch? > v=oIe5Zl2YsLE&feature=youtu.be > > > Awesome demo Steve, thanks for doing this. Particularly glad to see folks > using and extending the failure injection client. > > The HADOOP-13786 iteration turns on throttle event generation. All the new > committer stuff is ready for it, but all the existing S3A FS ops react to a > throttle exception by failing, when they need to just back off a bit. This > complicates testing as I have to explicitly turn off fault injection for > setup & teardown > > > Demoing the CLI tool was great as well. > > > I'm going to have to do another iteration on that CLI tool post-merge, as > I had one big problem: working out if the bucket and all the binding > settings meant it was "guarded". I think we'll need to track what issues > like that crop up in the field and add the diagnostics/other options. > > +I think another one that'd be useful would be to enum all s3guard DDB > tables in a region/globally & list their allocated IOPs. I know the AWS UI > can list tables by region, but you need to look around every region to find > out if you've accidentally created one. If you enum all table & look for a > s3guard version marker, then you can identify tables. > > Wanted to mention two things: > > 1. Authoritative mode is not fully implemented yet with Dynamo (it needs > to persist an extra bit for directories). I do have an auth-mode patch > (done for a hackathon) that I need to post which shows large performance > improvements over what S3Guard has today. As you said, we don't consider > authoritative mode ready for production yet: we want to play with it more > and improve the prune algorithm first. Authoritative mode can be thought > of as a nice bonus in the future: The main goal of S3Guard v1 is to fix the > get / list consistency issues you mentioned, which it does well. > > > we need to call that out in the release notes. > > 2. Also wanted to thank Lei (Eddy) Xu, he was very active during early > design and contributed some patches as well. > > > good point. Lei: you will get a special mention the next time I do the demo > > > Again, great demo, enjoyed it! > > -AF > > > its actually quite hard to show any benefits of s3guard on the command > line, so I've ended up showing some scala tests where I turn on the > (bundled) inconsistent AWS client to show how you then need to enable > s3guard to make the stack traces go away > > > On 22 Aug 2017, at 11:17, Steve Loughran <ste...@hortonworks.com<mailto: > ste...@hortonworks.com><mailto:ste...@hortonworks.com<mailto: > ste...@hortonworks.com>>> wrote: > > +1 (binding) > > I'm happy with it; it's a great piece of work by (in no particular order): > Chris Nauroth, Aaron Fabbri, Sean McRory & Mingliang Liu. plus a few bits > in the corners where I got to break things while they were all asleep. Also > deserving a mention: Thomas Demoor & Ewan Higgs @ WDC for consultancy on > the corners of S3, everyone who tested in (including our QA team), Sanjay > Radia, & others. > > I've already done a couple of iterations of fixing checksyles & code > reviews, so I think it is ready. I also have a branch-2 patch based on > earlier work by Mingliang, for people who want that. > > > > > On 17 Aug 2017, at 23:07, Aaron Fabbri <fab...@cloudera.com<mailto:fa > b...@cloudera.com><mailto:fab...@cloudera.com<mailto:fab...@cloudera.com>>> > wrote: > > Hello, > > I'd like to open a vote (7 days, ending August 24 at 3:10 PST) to merge the > HADOOP-13345 feature branch into trunk. > > This branch contains the new S3Guard feature which adds metadata > consistency features to the S3A client. Formatted site documentation can > be found here: > > https://github.com/apache/hadoop/blob/HADOOP-13345/ > hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md > > The current patch against trunk is posted here: > > https://issues.apache.org/jira/browse/HADOOP-13998 > > The branch modifies the s3a portion of the hadoop-tools/hadoop-aws module: > > - The feature is off by default, and care has been taken to insure it has > no impact when disabled. > - S3Guard can be enabled with the production database which is backed by > DynamoDB, or with a local, in-memory implementation that facilitates > integration testing without having to pay for a database. > - getFileStatus() as well as directory listing consistency has been > implemented and thoroughly tested, including delete tracking. > - Convenient Maven profiles for testing with and without S3Guard. > - New failure injection code and integration tests that exercise it. We > use timers and a wrapper around the Amazon SDK client object to force > consistency delays to occur. This allows us to assert that S3Guard works > as advertised. This will be extended with more types of failure injection > to continue hardening the S3A client. > > Outside of hadoop-tools/hadoop-aws's s3a directory there are some minor > changes: > > - core-default.xml defaults and documentation for s3guard parameters. > - A couple additional FS contract test cases around rename. > - More goodies in LambdaTestUtils > - A new CLI tool for inspecting and manipulating S3Guard features, > including the backing MetadataStore database. > > This branch has seen extensive testing as well as use in production. This > branch makes significant improvements to S3A's test toolkit as well. > > Performance is typically on par with, and in some cases better than, the > existing S3A code without S3Guard enabled. > > This feature was developed with contributions and feedback from many > people. I'd like to thank everyone who worked on HADOOP-13345 as well as > all of those who contributed feedback and work on the original design > document. > > This is the first major Apache Hadoop project I've worked on from start to > finish, and I've really enjoyed it. Please shout if I've missed anything > important here or in the VOTE process. > > Cheers, > Aaron Fabbri > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org<mailto: > common-dev-unsubscr...@hadoop.apache.org><mailto:common-dev- > unsubscr...@hadoop.apache.org<mailto:common-dev-unsubscribe@ > hadoop.apache.org>> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org<mailto: > common-dev-h...@hadoop.apache.org><mailto:comm > on-dev-h...@hadoop.apache.org<mailto:common-dev-h...@hadoop.apache.org>> > > > > >
