Hi, I have stopped working on s3n for a long time now. In case you are working with parquet and writing files s3a is the only alternative to failures. Otherwise why not use just s3://?
Regards, Gourav On Wed, Apr 13, 2016 at 12:17 PM, Steve Loughran <[email protected]> wrote: > > On 12 Apr 2016, at 22:05, Martin Eden <[email protected]> wrote: > > Hi everyone, > > Running on EMR 4.3 with Spark 1.6.0 and the provided S3N native driver I > manage to process approx 1TB of strings inside gzipped parquet in about 50 > mins on a 20 node cluster (8 cores, 60Gb ram). That's about 17MBytes/sec > per node. > > This seems sub optimal. > > The processing is very basic, simple fields extraction from the strings > and a groupBy. > > Watching Aaron's talk from the Spark EU Summit: > https://youtu.be/GzG9RTRTFck?t=863 > > it seems I am hitting the same issues with suboptimal S3 throughput he > mentions there. > > I tried different numbers of files for the input data set (more smaller > files vs less larger files) combined with various settings > for fs.s3n.block.size thinking that might help if each mapper streams > larger chunks. It didn't! It actually seems that many small files gives > better performance than less larger ones (of course with oversubscribed > number of tasks/threads). > > Similarly to what Aaron is mentioning with oversubscribed tasks/threads we > also become CPU bound (reach 100% cpu utilisation). > > > Has anyone seen a similar behaviour? How can we optimise this? > > Are the improvements mentioned in Aaron's talk now part of S3n or S3a > driver or are they just available under DataBricksCloud? How can we benefit > from those improvements? > > Thanks, > Martin > > P.S. Have not tried S3a. > > > > s3n is getting, deliberately, absolutely no maintenance except for > critical bug fixes. Every time something minor was done there (like bump up > a jets3t version), something else breaks. > > S3a is getting the work, and is the one you have to be using. > > It's also where the the performance analysis and enhancement is going on > > https://issues.apache.org/jira/browse/HADOOP-11694 > > a spark-related patch that's gone in is: > https://issues.apache.org/jira/browse/HADOOP-12810 ; significant > optimisation of time to enum lists of files in a partition; that could be > the root cause of a mentioned problem. Otherwise, a recent change is > HADOOP-12444, "Lazy-seek", where you can call seek() as much as you like, > but theres no attempt to open or close+reopen an HTTPS connection until the > next read. That really boosts performance on code doing > iostream.readFully(position, ... ), because the way readFully() was > implemented it was doing a > > long pos =getPos() > seek(newpos) > read(...) > seek(oldpos) > > Two seeks, even if they were reading in sequential blocks in the files... > this was hurting ORC reads where there's a lot of absolute reading and > skipping of sections of the file. > > There's also some ongoing work at the spark layer to do more in parallel: > https://github.com/apache/spark/pull/11242 > > If you look at the a lot of the work, it's coming from people (there > netflix), trying to understand why things are slow, and fixing them. > Anything other people can do to help here is welcome > > A key first step, even if you don't want to contribute any code back to > the OSS projects is : test other people's work. > > Anyone who can test the SPARK-9926 patch against their datasets should > apply that PR and test to see if shows (a) speedup, (b) no change or (c) > dramatic slowdown in performance. > > Similarly, the s3a performance working going in hadoop 2.8+ are things you > can test today, both by grabbing the branch and building spark against it, > or even going one step up and testing those patches before they get in. > > It's during the development time where any performance, functionality, > reliability problems can be fixed overnight —get testing with your code, to > know things will work when releases ship. This is particularly important > against object stores, because none of the Jenkins builds of the projects > test against S3. If you really care about S3 performance, and want to make > sure ongoing development is improving your ilfe: you need to get involved. > > -Steve > > > > >
