On 25 Aug 2017, at 20:22, Aaron Fabbri <fab...@cloudera.com<mailto:fab...@cloudera.com>> wrote:
Thank you everyone for reviewing and voting on the S3Guard feature branch merge. It looks like the Vote was a success. We have six binding +1's (Steve Loughran, Sean Mackrory, Mingliang Liu, Sanjay Radia, Kihwal Lee, and Lei (Eddy) Xu) and zero -1's. I will coordinate w/ Steve L to get this committed to trunk. I think we are going to bring it to branch-2 as well. -AF Update: this is now committed to trunk! This was a major piece of work —and it's been a great time working with people. Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu, Sean Mackrory, & others, as well as the effort of everyone who tested this, helped with the documentation, complained when it broke, etc. Special mention: Thomas Demoor & Ewan Higgs for explaining the low-level details of S3 protocols in a way that AWS themselves don't document. It's gone in as one big patch & not many small ones; we'd always planned it that way and had regularly merged trunk into the branch, its got regressions and fixes of them in. it'd be a mess. Now it's a single patch and you know who to complain to when it doesn't work. Sorry. What next in S3A land, well, let me see * HADOOP-14825 is where all the unfinished S3Guard work goes, with HADOOP-14220 being some CLI improvements I've been adding based on recent use. * HADOOP-13786 is my big "0-rename committer". It's been a regularly rebased branch atop the HADOOP-13345 branch, alongside an external downstream module to test the spark integration (we know the mapred v2 API stuff works, its only Spark & Parquet which doesn't play). * With S3Guard in, you can now turn on listing inconsistency in the client; in HADOOP-13786 I've added more fault injection in the form of "service throttled" responses. S3A doesn't handle them yet, which needs to be fixed not just in the new commit operations (which do), but in every single FileSystem API call. Same for other failures. Hence HADOOP-14531<https://issues.apache.org/jira/browse/HADOOP-14531> . As usual, people willing to code, document & test welcome. Go on, download trunk, test with s3guard enabled: now is the time to complain that things don't work! -Steve.
