Have you tried lowering mapred.job.reuse.jvm.num.tasks ? On Sun, Sep 26, 2010 at 3:30 AM, Bradford Stephens < [email protected]> wrote:
> Nope, that didn't seem to help. > > On Sun, Sep 26, 2010 at 1:00 AM, Bradford Stephens > <[email protected]> wrote: > > I'm going to try running it on high-RAM boxes with -Xmx4096m or so, > > see if that helps. > > > > On Sun, Sep 26, 2010 at 12:55 AM, Bradford Stephens > > <[email protected]> wrote: > >> Greetings, > >> > >> I'm running into a brain-numbing problem on Elastic MapReduce. I'm > >> running a decent-size task (22,000 mappers, a ton of GZipped input > >> blocks, ~1TB of data) on 40 c1.xlarge nodes (7 gb RAM, ~8 "cores"). > >> > >> I get failures randomly --- sometimes at the end of my 6-step process, > >> sometimes at the first reducer phase, sometimes in the mapper. It > >> seems to fail in multiple areas. Mostly in the reducers. Any ideas? > >> > >> Here's the settings I've changed: > >> -Xmx400m > >> 6 max mappers > >> 1 max reducer > >> 1GB swap partition > >> mapred.job.reuse.jvm.num.tasks=50 > >> mapred.reduce.parallel.copies=3 > >> > >> > >> java.lang.OutOfMemoryError: GC overhead limit exceeded > >> at java.nio.CharBuffer.wrap(CharBuffer.java:350) > >> at java.nio.CharBuffer.wrap(CharBuffer.java:373) > >> at > java.lang.StringCoding$StringDecoder.decode(StringCoding.java:138) > >> at java.lang.StringCoding.decode(StringCoding.java:173) > >> at java.lang.String.(String.java:443) > >> at java.lang.String.(String.java:515) > >> at > org.apache.hadoop.io.WritableUtils.readString(WritableUtils.java:116) > >> at > cascading.tuple.TupleInputStream.readString(TupleInputStream.java:144) > >> at > cascading.tuple.TupleInputStream.readType(TupleInputStream.java:154) > >> at > cascading.tuple.TupleInputStream.getNextElement(TupleInputStream.java:101) > >> at > cascading.tuple.hadoop.TupleElementComparator.compare(TupleElementComparator.java:75) > >> at > cascading.tuple.hadoop.TupleElementComparator.compare(TupleElementComparator.java:33) > >> at > cascading.tuple.hadoop.DelegatingTupleElementComparator.compare(DelegatingTupleElementComparator.java:74) > >> at > cascading.tuple.hadoop.DelegatingTupleElementComparator.compare(DelegatingTupleElementComparator.java:34) > >> at > cascading.tuple.hadoop.DeserializerComparator.compareTuples(DeserializerComparator.java:142) > >> at > cascading.tuple.hadoop.GroupingSortingComparator.compare(GroupingSortingComparator.java:55) > >> at > org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373) > >> at > org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:136) > >> at > org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103) > >> at > org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335) > >> at > org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350) > >> at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156) > >> at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2645) > >> at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2586) > >> > >> -- > >> Bradford Stephens, > >> Founder, Drawn to Scale > >> drawntoscalehq.com > >> 727.697.7528 > >> > >> http://www.drawntoscalehq.com -- The intuitive, cloud-scale data > >> solution. Process, store, query, search, and serve all your data. > >> > >> http://www.roadtofailure.com -- The Fringes of Scalability, Social > >> Media, and Computer Science > >> > > > > > > > > -- > > Bradford Stephens, > > Founder, Drawn to Scale > > drawntoscalehq.com > > 727.697.7528 > > > > http://www.drawntoscalehq.com -- The intuitive, cloud-scale data > > solution. Process, store, query, search, and serve all your data. > > > > http://www.roadtofailure.com -- The Fringes of Scalability, Social > > Media, and Computer Science > > > > > > -- > Bradford Stephens, > Founder, Drawn to Scale > drawntoscalehq.com > 727.697.7528 > > http://www.drawntoscalehq.com -- The intuitive, cloud-scale data > solution. Process, store, query, search, and serve all your data. > > http://www.roadtofailure.com -- The Fringes of Scalability, Social > Media, and Computer Science >
